The most important new feature added in this release is Hair Segmentation.
We have added the new type of tracking that allows to implement virtual
try-on effects like hair recoloring, patching of hairs in headwear
virtual try-on, and many more. Hair Segmentation is GPGPU powered and
real-time, it supports smooth visual effects on mobile devices.
The standard set of core tools to enable hair segmentation in an app:
Already available plugins implementing processings of segmentation masks
are compatible with hair segmentation mask by default. Developers can use
familiar building blocks to assemble pipelines of visually rich AR effects.
ShaderPlugin can be used as
a starting point of a custom visual effect on top of hair segmentation mask.
MaskBinaryPlugin
applying binarization operation to a segmentation mask. This plugin can
be used within mask processing pipeline to filter pixels based on their
probability and separate foreground and background pixels by a threshold.
MaskStepPlugin
is a similar but more advanced version of the plugin applying smoothstep
operation between two thresholds. Compared to more agressive Heaviside
step function used in simple binarization, smoothstep provides smooth
transition of mask value between foreground and background regions still
separating them. We recommend using MaskStepPlugin for smoother visuals.
MaskFilterPlugin
performs temporal filtering of a segmentation mask that is based on
minimization of entropy in probability space. It can be integrated
into a mask post-processing pipeline to decrease temporal noise.
One notable new plugin designed specifically for hair segmentation mask is
MaskSharpPlugin.
When used in post-processing pipeline of a segmentation mask it significantly
improves its accuracy. For example, in case of hair segmentation it can even
highlight separate locks of hairs that were missed by a neural network. We
recommend to apply a slight spacial smoothing (about 0.5) of a mask before
MaskSharpPlugin. If not provided by the Processor temporal smoothing of
a segmentation mask can also improve final results. While improvement of
mask quality is significant MaskSharpPlugin is really fast and can be used
in mobile browsers without sacrificing frames per second.
In this release we reimplemented body patching processing and corresponding plugins.
Partial body patching
is now not limited in radius of search for the closes body part defined by “patch”
and “keep” meshes. This means that no matter how long is the distance between a
pixel of a segmentation mask and the closest mesh it’ll be correctly classified as
“patch” or “keep” pixel. Moreover this stage of processing is now blazingly fast
and processing time doesn’t depend on a search radius. This makes usage of body
patching more convenient across all virtual try-on types. Additionally, the same
algorithm is applied within patching procedure itself improving it’s performance,
quality, and accuracy. The same improvements and changes are applied within
full body patching.
On top of that, to improve quality of image processing and overcome floating point
precision limits in shaders on mobile devices we implement multi-stage refinement of
results of intermediate processing stages. Additionally, memory access pattern is
optimized for all GPGPU kernels across Engeenee SDK.
Hand pose detection was optimized to provide more precise and smooth tracking.
We’ve implemented totally new approach for wrist detection that provides much
more robust and stable results and overcomes major disadvantages of the previous
algorithm. New wrist estimation borrows many principles from phalanx refinement
procedure introduced in the previous release and bringing promising results.
Wrist is approximated by an elliptical cylindroid with subpixel accuracy.
New detector has much better performance, it’s at least twice faster, thus
virtual try-on of watches and bracelets has gained significant boost in FPS.
On top of that, GPGPU kernels of wrist detector were significantly optimized
for real-time usage on mobile devices. Wrist detection can be enabled setting
corresponding parameter.
WristTrackPlugin
may be utilized to align scene node with a wrist estimation.
HandAlignPlugin
has been considerably reworked. Now we utilize minimization of back-projection
error when fitting hand armature in detected keypoints and iteratively converge
skeleton in approximated pose to keypoints at the same time preserving relative
lengths and scales. We utilized similar approach when aligning a body armature.
HandAlignPlugin still requires some work to be 100% ready for virtual try-on of
gloves, we are planning to complete implementation within the next release cycle.
This release introduces number of improvements in alignment of armature and
detected 3D pose including more natural kinematics, both forward and inverse.
We improved fitting of a skeleton into detected points relaxing some hard
requirements that when enforced lead to less natural deformations of a mesh.
Kinematics of shoulder bones has been reworked to provide less deformations
within shoulders and underarms areas. We also fixed rescaling issue near
shoulders using the same approach we implemented for limbs in the previous
version of the SDK.
Spine curve
parameter of ClothAlignPlugin has been fixed and when not specified or
set to null | undefined default 1.0 value is used. At the same time
0.0 value is now treated properly and disables curvature recovery.
Behaviour of spine bones is also improved to reduce unnatural rescaling.
ImageTexture supports generation
of mipmap levels via new convenient API. Mipmaps can considerably improve
performance of certain image processing algorithms or visual effect shaders.
Number of mipmap levels required to reach 1x1 resolution is evaluated
internally. Mipmaps are generated automatically on texture update or upload.
ShaderProgram provides an
option to opt-in for output texture with auto-generated mipmaps that can
be useful when chaining image processing stages.
New read() method
of a ShaderProgram copies GPU texture data to the host memory providing
access to raw pixel values.
Helper TextureReader class
provides functionality to get texture data on the client side and use
it within image processing pipelines. The important utility provided by
TextureReader is non-blocking asynchronous read of texture data using
readAsync().
In scenarios where data is not needed immediately or can be accessed with
a delay, usage of readAsync() can considerably improve performance as
blocking reads stall the whole GPU pipeline and wait until all operations
having the texture as a dependency are finished, this in turn significantly
decreases performance and utilization of a GPU device. Implementation of
readAsync() is thread-safe and correctly handles WebGL state recovery.
Generation of mipmaps and asynchronous read of textures allowed to reimplement
BrightnessPlugin and
derived LightsPlugin
and make them very fast. Now they can be used in rendering pipelines without
sacrificing frames per second. Additionally BrightnessPlugin was fine-tuned
to react on illumination changes mush faster.
We’ve improved API of video capture setup and made it more flexible. Previously,
we had two sets of capture options available. One option is to provide standard
MediaStreamConstraints defining set of device capabilities and allowing complete
control over selection of a video device. The second option is our simplified
VideoParams that provide basic
selection of a video device. The main downside was that advanced features like
rotation or cropping of the input video streams were available only through
VideoParams. To effectively resolve this limitation we’ve merged VideoParams
and MediaStreamConstraints options. Now fine-grained specification of the
device is available via opts
field of VideoParams. This allows for example to request specific deviceId
or even focus or white balance modes. Provided MediaStreamConstraints have
higher priority than the rest or parameters of VideoParams.
SDK users can provide custom options to underlying babylon.js or three.js
renderer when initializing
BabylonUniRenderer
or ThreeRenderer.
This might be useful when more fine-grained control over rendering pipeline
or canvas settings are required.
- Documentation page of Engeenee SDK is now generated by Starlight/Astro.
- Several fixes where made in saving state of rendering pipeline when
WebGL context is shared between several clients.
- User can provide custom options to underlying babylon.js or three.js
renderer when constructing BabylonUniRenderer or ThreeRenderer.
- Fine-tuning of temporal filter for 2D and 3D pose keypoints.
- Optimization of memory access in GPGPU kernels with performance improvement.
