Mathematics of 2D UI Elements in 3D Space: A Practical Guide

Working with 2D UI elements in 3D environments presents unique mathematical challenges. Through my contributions to three-mesh-ui and years of working with 3D graphics, I've learned that understanding the underlying mathematics is crucial for creating intuitive and performant 3D interfaces.

The Fundamental Challenge

When we place a 2D element (like a button or text label) in 3D space, we need to solve several mathematical problems:

Positioning: Where exactly should the element be placed in 3D coordinates?
Orientation: How should the element face the camera or user?
Scaling: How should the element size adapt to distance and perspective?
Interaction: How do we map 2D mouse/touch coordinates to 3D space?

Coordinate System Transformations

The foundation of 3D UI is understanding coordinate transformations. We typically work with multiple coordinate spaces:

javascript

1// World space coordinates (global 3D position)
2const worldPosition = new Vector3(5, 2, -3);
3
4// View space (relative to camera)
5const viewMatrix = camera.matrixWorldInverse;
6const viewPosition = worldPosition.clone().applyMatrix4(viewMatrix);
7
8// NDC (Normalized Device Coordinates) - Three.js .project() returns NDC
9const ndcPosition = worldPosition.clone().project(camera);
10
11// Pixel space conversion helper
12function ndcToPixels(ndc, canvas) {
13  return {
14    x: ((ndc.x + 1) * canvas.width) / 2,
15    y: ((1 - ndc.y) * canvas.height) / 2,
16  };
17}

The Projection Matrix

The key to understanding 3D UI positioning lies in the projection transformation. Note: Three.js provides vector.project(camera) for this, but understanding the math is valuable:

javascript

1// Custom projection implementation - returns NDC with view-space depth
2// For production code, use Three.js's built-in vector.project(camera)
3function projectPointToNDC(point3D, camera) {
4  const fov = (camera.fov * Math.PI) / 180; // Convert to radians
5  const aspect = camera.aspect;
6
7  // Transform to view space
8  const viewPoint = point3D.clone().applyMatrix4(camera.matrixWorldInverse);
9
10  // Apply perspective division to get NDC coordinates
11  const projected = {
12    x: ((viewPoint.x / -viewPoint.z) * (1 / Math.tan(fov / 2))) / aspect,
13    y: (viewPoint.y / -viewPoint.z) * (1 / Math.tan(fov / 2)),
14    z: viewPoint.z, // Keep view-space depth for distance calculations
15  };
16
17  // Return NDC coordinates (-1 to 1) with view-space depth
18  return {
19    x: projected.x,
20    y: projected.y,
21    viewDepth: -viewPoint.z, // Positive distance from camera
22  };
23}
24
25// For practical use, prefer Three.js built-in method:
26// const ndcCoords = worldPosition.clone().project(camera)

Billboard Matrices: Always Facing the Camera

One common requirement is making 2D elements always face the camera (billboarding). Here's the mathematical approach:

javascript

1// Custom billboard implementation - for production, consider using camera.quaternion
2function createBillboardMatrix(elementPosition, cameraPosition, cameraUp) {
3  // Calculate the direction from element to camera
4  const forward = new Vector3()
5    .subVectors(cameraPosition, elementPosition)
6    .normalize();
7
8  // Calculate right vector (cross product of world up and forward)
9  const right = new Vector3().crossVectors(cameraUp, forward).normalize();
10
11  // Calculate actual up vector (cross product of forward and right)
12  const up = new Vector3().crossVectors(forward, right).normalize();
13
14  // Create the billboard matrix - basis vectors go in COLUMNS
15  const billboardMatrix = new Matrix4().set(
16    right.x,
17    up.x,
18    forward.x,
19    elementPosition.x,
20    right.y,
21    up.y,
22    forward.y,
23    elementPosition.y,
24    right.z,
25    up.z,
26    forward.z,
27    elementPosition.z,
28    0,
29    0,
30    0,
31    1
32  );
33
34  return billboardMatrix;
35}
36
37// Alternative: Use Three.js built-in approach
38// object.lookAt(camera.position)
39// or copy camera rotation: object.quaternion.copy(camera.quaternion)

Distance-Based Scaling

UI elements should often scale based on their distance from the camera to maintain consistent apparent size:

javascript

1function calculateUIScale(elementPosition, camera, baseScale = 1) {
2  // Use view-space depth for accurate scaling
3  const viewPosition = elementPosition
4    .clone()
5    .applyMatrix4(camera.matrixWorldInverse);
6  const viewDepth = -viewPosition.z; // Positive distance from camera
7  const referenceDepth = 10; // Distance at which scale = baseScale
8
9  // Inverse scaling - UI elements get smaller with distance
10  const perspectiveScale = baseScale * (referenceDepth / viewDepth);
11
12  // Alternative: Constant pixel size scaling using vFOV
13  const fovScale = 2 * Math.tan((camera.fov * Math.PI) / 360); // vFOV factor
14  const constantPixelScale =
15    baseScale * (referenceDepth / viewDepth) * fovScale;
16
17  // Logarithmic scaling for more gradual changes
18  const logScale =
19    (baseScale * Math.log(referenceDepth + 1)) / Math.log(viewDepth + 1);
20
21  return perspectiveScale; // Use inverse scaling for natural UI behavior
22}

Ray Casting for 3D Interaction

Converting 2D mouse coordinates to 3D interactions requires ray casting mathematics:

javascript

1// Custom raycasting implementation
2// For production, use THREE.Raycaster which handles both camera types
3function mouseToRay(mouseX, mouseY, camera, canvas) {
4  // Normalize mouse coordinates to -1 to 1 range
5  const normalizedX = (mouseX / canvas.width) * 2 - 1;
6  const normalizedY = -(mouseY / canvas.height) * 2 + 1;
7
8  if (camera.type === "PerspectiveCamera") {
9    // Perspective camera - ray from camera position through screen point
10    const rayOrigin = camera.position.clone();
11    const rayDirection = new Vector3(normalizedX, normalizedY, -1)
12      .unproject(camera)
13      .sub(rayOrigin)
14      .normalize();
15
16    return { origin: rayOrigin, direction: rayDirection };
17  } else {
18    // Orthographic camera - parallel rays
19    const rayDirection = new Vector3(0, 0, -1).transformDirection(
20      camera.matrixWorld
21    );
22
23    const rayOrigin = new Vector3(normalizedX, normalizedY, 0).unproject(
24      camera
25    );
26
27    return { origin: rayOrigin, direction: rayDirection };
28  }
29}
30
31// Recommended approach using Three.js built-in Raycaster:
32// const raycaster = new THREE.Raycaster()
33// raycaster.setFromCamera(normalizedMouse, camera)
34// const intersects = raycaster.intersectObjects(objects)
35
36function rayPlaneIntersection(ray, planeNormal, planePoint) {
37  const denom = planeNormal.dot(ray.direction);
38
39  if (Math.abs(denom) < 1e-6) {
40    return null; // Ray is parallel to plane
41  }
42
43  const t = planePoint.clone().sub(ray.origin).dot(planeNormal) / denom;
44
45  if (t < 0) {
46    return null; // Intersection is behind the ray origin
47  }
48
49  return ray.origin.clone().add(ray.direction.clone().multiplyScalar(t));
50}

Quaternion Rotations for Smooth Orientation

When animating UI element orientations, quaternions provide smooth interpolation:

javascript

1function createOrientationQuaternion(fromDirection, toDirection) {
2  // Normalize input vectors
3  const from = fromDirection.clone().normalize();
4  const to = toDirection.clone().normalize();
5
6  // Calculate rotation quaternion between two directions
7  const quaternion = new Quaternion();
8  quaternion.setFromUnitVectors(from, to);
9
10  return quaternion;
11}
12
13// Alternative: Full "look rotation" that honors up vector
14function createLookRotation(forward, up = new Vector3(0, 1, 0)) {
15  const f = forward.clone().normalize();
16  const r = new Vector3().crossVectors(up, f).normalize();
17  const u = new Vector3().crossVectors(f, r).normalize();
18
19  const matrix = new Matrix4().set(
20    r.x,
21    u.x,
22    f.x,
23    0,
24    r.y,
25    u.y,
26    f.y,
27    0,
28    r.z,
29    u.z,
30    f.z,
31    0,
32    0,
33    0,
34    0,
35    1
36  );
37
38  return new Quaternion().setFromRotationMatrix(matrix);
39}
40
41function smoothOrientationTransition(
42  currentQuat,
43  targetQuat,
44  deltaTime,
45  speed = 2.0
46) {
47  // Spherical linear interpolation (SLERP)
48  const t = Math.min(1.0, deltaTime * speed);
49  return currentQuat.clone().slerp(targetQuat, t);
50}

Practical Implementation: Text Labels in 3D

Here's how these concepts come together in a practical text label system:

javascript

1class TextLabel3D {
2  constructor(text, position, options = {}) {
3    this.text = text;
4    this.worldPosition = position.clone();
5    this.options = {
6      baseScale: 1,
7      billboard: true,
8      distanceScaling: true,
9      ...options,
10    };
11
12    this.element = this.createElement();
13    this.transform = new Matrix4();
14  }
15
16  updateTransform(camera) {
17    let scale = this.options.baseScale;
18
19    // Apply distance-based scaling
20    if (this.options.distanceScaling) {
21      scale *= calculateUIScale(this.worldPosition, camera);
22    }
23
24    // Create transform matrix
25    this.transform.identity();
26
27    if (this.options.billboard) {
28      // Method 1: Custom billboard matrix
29      const billboardMatrix = createBillboardMatrix(
30        this.worldPosition,
31        camera.position,
32        camera.up
33      );
34      this.transform.multiplyMatrices(billboardMatrix, this.transform);
35
36      // Method 2: Use Three.js built-in (recommended)
37      // this.transform.setPosition(this.worldPosition)
38      // this.transform.lookAt(camera.position)
39      // or: this.transform.extractRotation(camera.matrixWorld)
40    } else {
41      this.transform.setPosition(this.worldPosition);
42    }
43
44    // Apply scaling
45    this.transform.scale(new Vector3(scale, scale, scale));
46  }
47
48  isVisible(camera) {
49    // Basic frustum culling using Three.js Frustum
50    const frustum = new Frustum();
51    frustum.setFromProjectionMatrix(
52      new Matrix4().multiplyMatrices(
53        camera.projectionMatrix,
54        camera.matrixWorldInverse
55      )
56    );
57
58    // For point-based culling
59    return frustum.containsPoint(this.worldPosition);
60
61    // Alternative: Simple NDC bounds check (less accurate)
62    // const ndcPos = this.worldPosition.clone().project(camera)
63    // return ndcPos.x >= -1 && ndcPos.x <= 1 &&
64    //        ndcPos.y >= -1 && ndcPos.y <= 1 &&
65    //        ndcPos.z >= -1 && ndcPos.z <= 1
66  }
67}

Performance Considerations

When working with many 3D UI elements, consider these optimizations:

Frustum Culling: Only update elements visible to the camera
Level of Detail: Use simpler representations for distant elements
Batching: Group similar elements to reduce draw calls
Occlusion: Hide elements blocked by 3D geometry

javascript

1// Proper frustum culling using Three.js built-ins
2function isInCameraFrustum(position, camera, margin = 0.0) {
3  const frustum = new Frustum();
4  frustum.setFromProjectionMatrix(
5    new Matrix4().multiplyMatrices(
6      camera.projectionMatrix,
7      camera.matrixWorldInverse
8    )
9  );
10
11  return frustum.containsPoint(position);
12}
13
14// For bounding volumes (more accurate for complex objects)
15function isBoundingBoxVisible(boundingBox, camera) {
16  const frustum = new Frustum();
17  frustum.setFromProjectionMatrix(
18    new Matrix4().multiplyMatrices(
19      camera.projectionMatrix,
20      camera.matrixWorldInverse
21    )
22  );
23
24  return frustum.intersectsBox(boundingBox);
25}

Key Takeaways

Master coordinate transformations - Understanding world, view, and screen spaces is fundamental
Use quaternions for rotations - They prevent gimbal lock and provide smooth interpolation
Implement proper scaling - Distance-based scaling maintains visual consistency
Optimize for performance - Use frustum culling and batching for complex scenes
Test across viewing angles - 3D UI behaves differently from various camera positions

The mathematics behind 3D UI might seem complex, but these principles form the foundation for creating intuitive and performant three-dimensional interfaces. Understanding these concepts has been crucial in my work with three-mesh-ui and other 3D interface projects.

Important Note: While understanding these mathematical foundations is valuable, Three.js provides optimized, battle-tested implementations of most of these concepts. For production applications, leverage built-in methods like vector.project(), Raycaster, Frustum, and object.lookAt() whenever possible. The custom implementations shown here are educational tools to understand the underlying mathematics.

Working on a 3D interface project? Let's discuss how these mathematical principles can be applied to your specific use case.