Repository navigation
Implement iOS VoiceOver support - #18016
Conversation
|
You can test this PR using the following package version. |
|
You can test this PR using the following package version. |
|
You can test this PR using the following package version. |
|
You can test this PR using the following package version. |
|
You can test this PR using the following package version. |
|
You can test this PR using the following package version. |
|
You can test this PR using the following package version. |
|
You can test this PR using the following package version. |
|
You can test this PR using the following package version. |
|
You can test this PR using the following package version. |
|
You can test this PR using the following package version. |
|
You can test this PR using the following package version. |
|
You can test this PR using the following package version. |
|
You can test this PR using the following package version. |
|
You can test this PR using the following package version. |
|
You can test this PR using the following package version. |
|
You can test this PR using the following package version. |
|
You can test this PR using the following package version. |
|
I don't think my code caused those tests to fail... anyone know if there's a known CI issue that could have caused it? |
|
You can test this PR using the following package version. |
|
You can test this PR using the following package version. |
|
I finally tested this branch on a simulator using Accessibility Inspector and have several questions:
A little rant: it's frustrating how sometimes difficult it is to get mobile workloads working after XCode update, and almost every time it's easier with vscode than rider. |
| MathUtilities.IsZero(bounds.Clip.Width) || | ||
| MathUtilities.IsZero(bounds.Clip.Height), | ||
| _ => !Owner.IsVisible, | ||
| _ => !Owner.IsEffectivelyVisible, |
There was a problem hiding this comment.
cc @grokys, is it fine to switch to Effectively Visible? Or do other backends handle inherited visibility on their own?
There was a problem hiding this comment.
Other backends handle this only in-so-far as they handle the hit testing in Avalonia code. This is in contrast to iOS which is the only target platform that performs the hit testing on the OS side. Therefore, I argue that this adaptation is necessary.
| foreach (AutomationPeer child in peer.GetChildren()) | ||
| { | ||
| AutomationPeerWrapper? wrapper; | ||
| if (!_childrenMap.TryGetValue(child, out wrapper) && | ||
| (child.GetName().Length > 0 || child.IsKeyboardFocusable())) | ||
| { | ||
| _childrenList.Add(child); | ||
| _childrenMap.Add(child, new(this, child)); | ||
| } | ||
|
|
||
| wrapper?.UpdatePropertiesIfValid(); | ||
| wrapper?.UpdateTraits(); | ||
|
|
||
| UpdateChildren(child); | ||
| } | ||
| } |
There was a problem hiding this comment.
I am not familiar with iOS accessibility, but shouldn't this backend map avalonia automation onto accessibilityContainer?
In other words, it should have a tree of peer wrappers, implementing IUIAccessibilityContainer, on wrapping children only on request. Instead of processing every element in the app into a single dictionary, updating it on each accessibility request. Otherwise, it needs to be carefully tested for performance and memory leaks, and potentially that problem with TabControl I mentioned above. At the moment, cleanup is only happening on GetAccessibilityElementAt request for a specific element, and if iOS doesn't call this method, no element is cleared, holding it in the memory.
By implementing IUIAccessibilityContainer on every node it would make it possible to properly handle accessibilityContainerType.
There was a problem hiding this comment.
The issue I encountered trying to implement what you're describing is that iOS then doesn't traverse the tree when using touch exploration. In other words, it ends up just settling on the root element because, like you are saying, it doesn't traverse deeper due to the OS not requesting that traversal in the first place.
Unless I were to do some kind of implicit aggregation in the accessibilityContainer methods that does this traversal for the OS (which would again bring up the performance concerns), this is the only way I was able to get touch exploration to (mostly) work.
All this said, I can definitely double check the cleanup code to make sure that we don't get the weird TabControl behavior you're describing.
|
Thanks @maxkatz6 for the in-depth code review. Below are responses to your comments in numerical order:
|
There was a problem hiding this comment.
Pull request overview
Adds iOS VoiceOver integration by bridging Avalonia automation peers to UIKit accessibility APIs.
Changes:
- Adds iOS accessibility containers, traits, actions, and provider mappings.
- Improves automation naming and effective visibility handling.
- Updates visibility tests and documentation.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
tests/Avalonia.Base.UnitTests/VisualTests.cs |
Updates detached-visibility expectations. |
src/iOS/Avalonia.iOS/AvaloniaView.cs |
Initializes the accessibility bridge. |
src/iOS/Avalonia.iOS/AvaloniaView.Automation.cs |
Exposes UIKit container callbacks. |
src/iOS/Avalonia.iOS/AutomationPeerWrapper.cs |
Implements VoiceOver peer wrapping and actions. |
src/Avalonia.Controls/Automation/Peers/ControlAutomationPeer.cs |
Uses effective visibility for off-screen state. |
src/Avalonia.Controls/Automation/Peers/ContentControlAutomationPeer.cs |
Propagates child names and help text. |
src/Avalonia.Controls/Automation/IsOffscreenBehavior.cs |
Updates visibility documentation. |
src/Avalonia.Base/Visual.cs |
Marks detached visuals effectively invisible. |
Suppressed comments (2)
src/iOS/Avalonia.iOS/AutomationPeerWrapper.cs:210
AccessibilityFramerequires screen coordinates, but the iOSPointToScreenimplementation currently returns the root-local point unchanged (AvaloniaView.cs:226). WhenAvaloniaViewis embedded anywhere other than the screen origin, every VoiceOver focus rectangle and hit target is offset by the view's window position. Convert the root-local rectangle through the native view/window, or expose it via the appropriate container-space frame property.
PixelRect screenRect = new PixelRect(
root?.PointToScreen(bounds.TopLeft) ?? default,
root?.PointToScreen(bounds.BottomRight) ?? default
);
CGRect nativeRect = new CGRect(
screenRect.X, screenRect.Y,
screenRect.Width, screenRect.Height
src/iOS/Avalonia.iOS/AutomationPeerWrapper.cs:236
- Toggle providers are actionable below, but their
ToggleStateis never represented inAccessibilityValue. A checkbox therefore has the same VoiceOver state before and afterToggle(), andToggleStatePropertyevents are also ignored by the setter map. Expose the toggle value (including indeterminate state) in the iOS representation and mapTogglePatternIdentifiers.ToggleStatePropertyso it refreshes after changes.
private static void UpdateValue(AutomationPeerWrapper self)
{
AutomationPeer peer = self;
string? newValue =
peer.GetProvider<IRangeValueProvider>()?.Value.ToString("0.##") ??
peer.GetProvider<IValueProvider>()?.Value;
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
You can test this PR using the following package version. |
|
@jmacato All looks good, the changes you committed are functional on a real iOS device. I also just now addressed the comments by Copilot. Cheers! |
|
You can test this PR using the following package version. |
|
Let's get this show on the road, merging in a bit! Just need to double check some stuff locally |
jmacato
left a comment
jmacato
left a comment
There was a problem hiding this comment.
Thank you very much for your work on this @IsaMorphic ! Avalonia's a11y story is gonna be more fantastic going forward!
|
Thanks so much for implementing this and keeping on top of it @IsaMorphic ! |
|
You're very welcome! As a sibling of visually impaired people, it's important to me that they get to enjoy Avalonia apps, too :) consider hiring me, haha! |
|
You can test this PR using the following package version. |
|
@IsaMorphic Thank you for including support for AutomationId in your PR — I can confirm it works. |
|
Out of curiosity, considering Android was backported to version 12.1.1, do you plan on doing the same for iOS or will it have to wait for a later version @jmacato ? Thank you again for this contribution @IsaMorphic 🙂 |
|
@NicholasLachapelle It's @IsaMorphic that needs to be thanked for this effort :D But I'll have to consult with the others if this can be backported. |
|
You're all so kind! :D I'm happy to have this finally committed! |
|
As a dev working on applications that very much care about accessibility and a blind iOS user besides, this is great news! Sooner we can get this the better :) |
@jmacato If needed I can take point on adapting this patch for a backport. This is a bugfix, not a feature! :) |
MrJul
added
the
backport-candidate-12.1.x
* Implement iOS VoiceOver support * Some investigation (it is not working) * Progress * Functional prototype for iOS VoiceOver support * Improved VoiceOver control function * Implement traits and accessibility actions for VoiceOver * Remove unused using statement * Default IsOffscreenBehavior should be based on IsEffectivelyVisible * Use most specific activation function first * Ensure that offscreen UIAccessibilityElements are removed * Fix interrupted speech upon focusing element * Accessibility improvements based on tests with low vision users * More improvements to iOS VoiceOver access * Final fixes for iOS VoiceOver * Add informative comments * Implement review suggestions for iOS VoiceOver * Fixed bugs in determining whether controls are offscreen * Ensure that AutomationPeer children are updated * Fix failing tests for IsEffectivelyVisible & CompositorHitTesting * Implement AccessibilityContainer for all peer wrappers Addresses performance concerns regarding the accessibility tree * Fix iOS VoiceOver not giving touch feedback for some labeled elements * More garbage collection for iOS VoiceOver * Consolidate garbage collection code for iOS VoiceOver feature * Final fixes for new iOS VoiceOver implementation * Pass informative parameters to UIAccessibility notifications * Fix crashing bug with iOS VoiceOver scroll gesture * Fix: use `IInputRoot.RootElement` to compute screen coordinates * Fix: use more precise logic for distinguishing groups from controls * Fix: iOS VoiceOver recognizes all container elements * Fix: recognize `SplitButton` as container type * Fix: include custom controls under container category * Fix: recognize commonly templated containers * Fix: use `HashSet` for determining whether control type maps to a container * Fix: optimizations for iOS VoiceOver * Fix: update accessibility properties unconditionally * Fix: use cached state for containers * Fix: use basic `HashSet` type for container type lookup * Fix: include `ScrollViewer` as group control type * Fix: remove references to `ImmutableDictionary` * Fix: VoiceOver should read page headers * Fix: implement support for `AutomationId` for iOS automation tree * Fix iOS accessibility peer updates * Fix: include recommended fixes and changes from Copilot review * Fix: revert formatting and modified tests * Fix iOS accessibility trait and view disposal --------- Co-authored-by: IsaMorphic <me@isamorphic.dev> Co-authored-by: Jumar Macato <16554748+jmacato@users.noreply.github.com>
MrJul
added
backported-12.1.x
and removed
backport-candidate-12.1.x
What does the pull request do?
This is a twin PR to #17704 that implements support for VoiceOver on iOS devices. Importantly, the UIAccessibility & UIAccessibilityContainer informal protocols are leveraged to create cohesion between Avalonia's automation system and that of the iOS ecosystem. Fixes #20125.
What is the current behavior?
Currently, Avalonia apps targeting the iOS platform do not expose user controls to the operating system's accessibility pipeline. This makes it impossible for blind and visually impaired users to use those apps with a screen reader or braille display.
What is the updated/expected behavior with this PR?
This PR enables users reliant on VoiceOver features to properly navigate Avalonia's view heirarchy using the appropriate, well-known shortcuts and gestures.
How was the solution implemented (if it's not obvious)?
The PR implements support for VoiceOver via Avalonia's
AutomationPeerAPI by wrapping those instances in a subclass ofUIAccessibilityElementthat implements all the necessary formal and informal protocols to create a fully realized experience for low vision users.Checklist