We’ve updated all packages used internally to the latest versions.
This release doesn’t introduce any API or logic changes, as well
as updates within body tracking or image/video post-processing.
Basically, only build system was updated: esbuild and rollup are
bumped to the latest versions, C++ modules implementing heavy math
computations switched to C++17 from C++14, and we’ve updated their
native dependencies as well.
One can migrate from the previous version of the SDK without any code
changes. All examples are updated to the 0.8.6 version. They can be
found on the getting started page.
To provide more efficient and optimized WASM modules that perform
all heavy computations in body tracking pipelines we’ve switched
to the newer version of the Emscripten SDK. More recent versions of
Emscripten dropped support of ES2017 and now require some features
from ES2020. Therefore, starting from this release we are bundling
all packages of the SDK in ES2020 modules. Required ES2020 features
are widely available across major browsers so there shouldn’t be
any compatibility issues when switching to the newer ECMAScript.
In this release we introduce the concept of priority score for tracking targets.
During the detection phase of a tracking pipeline several regions of interest
passing confidence threshold may be detected. Number of targets selected for
further tracking is limited by computational resources of a user’s device. Web
runtime can track only 1 target because of performance limitation of browsers,
especially on mobile. By default region of interest having the highest confidence
score is chosen for tracking.
We’ve added a set of utilities that make process of target selection more flexible
and controllable outside of Engeenee SDK. Namely, processors now have additional
optional parameter - rate function that evaluates measure of priority for detected
regions of interest. Detections with higher rates will be considered first for
further tracking. For example, these methods may guide selection of tracking targets
to prioritize bigger or more centered RoIs, downweight score according to some rules.
Default implementation forwards confidence score as rate. Targets with rate <= 0.0
will be completely ignored by tracking pipeline.
PoseProcessor can tune target
selection using rateRoI()
parameter. It is called for every
detected target and returns
its priority score. Tracking pipeline will sort RoIs by evaluated priority scores,
filter out ones that have <= 0, and choose the top N (1 for web) RoIs for tracking.
RoIs are usually defined in normalized coordinates, therefore callback has aspect
ration as a parameter, so implementation can use it to calculate correct measures.
FaceProcessor can tune target
selection using rateRoI()
parameter. It is called for every
detected target and returns
its priority score. Tracking pipeline will sort RoIs by evaluated priority scores,
filter out ones that have <= 0, and choose the top N (1 for web) RoIs for tracking.
RoIs are usually defined in normalized coordinates, therefore callback has aspect
ration as a parameter, so implementation can use it to calculate correct measures.
Babylon.js had an issue where setting rotation or rotationQuaterion
on a camera in right-handed coordinate system actually sets the camera
180 degrees flipped from what is represented in the view matrix. You
can read more in the description of
corresponding PR
fixing the issue. In Engeenee SDK we had a workaround for this. Now
when issue is fixed we’ve rolled back this workaround.
The major downside is that compatibility between different versions of
Engeenee SDK and Babylon.js became slightly more complicated with the
current release. If you are using a version of
Engeeneee prior to 0.8.2 please use Babylon.js version 8.9.1
or lower. Starting from version 0.8.2 the minimum compatible version
of Babylon.js is 8.11.0.
Babylon.js is a peer dependency of Engeenee’s
@geenee/bodyrenderers-babylon,
the rule of thumb when updating the SDK is to switch to the minimum
compatible version of Babylon.js first (the one stated in package.json
of bodyrenderers-babylon), and then gradually update the renderer if newer
features or fixes are required.
With this release we are switching to WebGL2 in all stages of processing
and rendering pipelines. Previously, processing and effect shaders of the
Engeenee SDK where written in WebGL1 for better compatibility with older
devices and browsers. Meanwhile, inference of neural networks and most
rendering engines already heavily utilize modern WebGL2 features that are
available in all browsers for at least 5 years. Therefore, we’ve decided
to port rest of GPGPU and image processing to the newer graphics standard
that is by the way already pretty old (=.
All shaders were re-written and porting process is very straightforward
and simple. If you have custom shaders utilized in
ShaderProgram or
ShaderPlugin,
please update the code to follow WebGL2 shader syntax. We recommend
this guide
that describes major differences between standards. Basically, one
needs to make the next simple changes to a shader code:
Replace attribure -> in
In vertex shader, replace varying -> out
In fragment shader, replace varying -> in
Replace gl_FragColor with any out vec4 variable and assign to it
Use texture() instead of texture2D(), textureCube(), etc
ShaderProgram adds #version 300 es line automatically
For fast prototyping of simple UIs in demos and examples we are using very
lightweight library of UI components developed internally. Gluon library
is build on top of amazing VanJS - ultra-lightweight,
zero-dependency, and unopinionated Reactive UI framework written in vanilla
JavaScript and DOM. VanJS doesn’t rely on JSX or introduce any special syntax
to compose UI components, therefore we don’t need any additional transpilers.
For styling Gluon utilizes UnoCSS - fast and flexible
atomic CSS engine.
With this release we are making Gluon library available for all users of the
Engeenee SDK. Engeenee remains framework agnostic and doesn’t restrict choice
of UI framework. We just think that very lightweight UI library having zero
dependencies and very easy to pick up may be useful as a starting point for
enthusiasts who would like to achieve the fastest performance possible, have
closer to bare-metal experience and concentrate more on the most interesting
parts: 3D, visuals, and interactivity.
