Foreign app navigation with Android Accessibility Services

Android Accessibility Services are intended to assist users with disabilities. Using an accessibility service you may for example change the input focus, scroll a list or press a button. Interestingly this can be done not only for your own app, but for any app on the device. Of course your app must request permission for this.

Scenario

This article describes the usage of accessibility services for the following sample use case:

1. Open calculator app
2. Press button «1»
3. Press button «+»
4. Press button «2»
5. Press button «=»
6. Read the result
7. Go back and display result

Although this is not really the intended use of an accessibility service, this simple use case should demonstrate the functionality in an easy way.

Service configuration

1. Configure accessibility service
   In a separate XML file we need to configure our accessibility service. Here we need to declare that our service wants to retrieve the active window content. The file should be placed in the resource directory of the application (<project_dir>/res/xml/myaccessibilityservice.xml).

   <?xml version="1.0" encoding="utf-8"?>
   <accessibility-service
       xmlns:android="http://schemas.android.com/apk/res/android"
       android:canRetrieveWindowContent="true" />
   

2. Edit AndroidManifest.xml
   Our accessibility service must be declared in the application element of the manifest. In the meta-data tag we reference the previously created accessibility service configuration.

   ...
   <service android:name="MyAccessibilityService" android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE">
       <intent-filter>
           <action android:name="android.accessibilityservice.AccessibilityService" />
       </intent-filter>
       <meta-data android:name="android.accessibilityservice" android:resource="@xml/myaccessibilityservice" />
   </service>
   ...
   

Implementation

MyActivity.java

From the main activity (MyActivity.java) we just start the calculator demo (CalculatorDemo.java)..

...
private final CalculatorDemo mCalculatorDemo;

public MyActivity() {
    mCalculatorDemo = new CalculatorDemo(this);
}
...
public void handleStartButtonClick(View view) {
    mCalculatorDemo.run();
}
...

MyAccessibilityService.java

To implement our accessibility service (MyAccessibilityService.java) we need to derive from the abstract class AccessibilityService. For the calculator demo we just override the onServiceConnected method in order to get a reference of the accessibility service. For the sake of simplicity we just inject it directly to our CalculatorDemo class.

...
public class MyAccessibilityService extends AccessibilityService {
    ...
    @Override
    protected void onServiceConnected() {
        super.onServiceConnected();
        CalculatorDemo.initialize(this);
    }
    ...
}

CalculatorDemo.java

Finally the main functionality is implemented in CalculatorDemo.java.

Using the method getRootInActiveWindow of the accessibility service we can access the view hierarchy of the current window. The view hierarchy is represented as a hierarchy of AccessibilityNodeInfo instances. We use the method findAccessibilityNodeInfosByViewId to find the relevant child nodes.

private static AccessibilityNodeInfo findNodeInCurrentWindow(String id) {
    AccessibilityNodeInfo window = mAccessibilityService.getRootInActiveWindow();
    return window.findAccessibilityNodeInfosByViewId(id).get(0);
}

Relevant nodes (and view ID’s) can be found by traversing the node hierarchy. Once we have the relevant nodes, actions can be performed by calling performAction on the specific node.

public void run() {

    // Start calculator app
    mContext.startActivity(new Intent().setClassName(PACKAGE, CLASS));
    sleep(2000);

    // Press button "1"
    findNodeInCurrentWindow(ID_BUTTON_01).performAction(AccessibilityNodeInfo.ACTION_CLICK);
    sleep(500);

    // Press button "+"
    findNodeInCurrentWindow(ID_BUTTON_ADD).performAction(AccessibilityNodeInfo.ACTION_CLICK);
    sleep(500);

    // Press button "2"
    findNodeInCurrentWindow(ID_BUTTON_02).performAction(AccessibilityNodeInfo.ACTION_CLICK);
    sleep(500);

    // Press button "="
    findNodeInCurrentWindow(ID_BUTTON_EQUAL).performAction(AccessibilityNodeInfo.ACTION_CLICK);
    sleep(500);

    // Read result
    String result = findNodeInCurrentWindow(ID_TEXT_DISPLAY).getText().toString();
    Toast.makeText(mContext, String.format("Result is %s", result), Toast.LENGTH_SHORT).show();

    // Go back
    mAccessibilityService.performGlobalAction(AccessibilityService.GLOBAL_ACTION_BACK);
}

Full implementation

The following code shows the full implementation of all relevant parts for this demo.

import android.os.Bundle;
import android.support.v7.app.AppCompatActivity;
public class MyActivity extends AppCompatActivity {

    private final CalculatorDemo mCalculatorDemo;

    public MyActivity() {
        mCalculatorDemo = new CalculatorDemo(this);
    }

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.myactivity);
    }

    public void handleStartButtonClick(View view) {
        mCalculatorDemo.run();
    }
}

import android.accessibilityservice.AccessibilityService;
import android.view.accessibility.AccessibilityEvent;

public class MyAccessibilityService extends AccessibilityService {

    @Override
    public void onAccessibilityEvent(AccessibilityEvent event) {
    }

    @Override
    public void onInterrupt() {
    }

    @Override
    protected void onServiceConnected() {
        super.onServiceConnected();
        CalculatorDemo.initialize(this);
    }
}

import android.accessibilityservice.AccessibilityService;
import android.content.Context;
import android.content.Intent;
import android.support.annotation.NonNull;
import android.view.accessibility.AccessibilityNodeInfo;
import android.widget.Toast;

public class CalculatorDemo {

    private static final String PACKAGE = "com.sec.android.app.popupcalculator";
    private static final String CLASS = "com.sec.android.app.popupcalculator.Calculator";
    private static final String ID_BUTTON_01 = "com.sec.android.app.popupcalculator:id/bt_01";
    private static final String ID_BUTTON_02 = "com.sec.android.app.popupcalculator:id/bt_02";
    private static final String ID_BUTTON_ADD = "com.sec.android.app.popupcalculator:id/bt_add";
    private static final String ID_BUTTON_EQUAL = "com.sec.android.app.popupcalculator:id/bt_equal";
    private static final String ID_TEXT_DISPLAY = "com.sec.android.app.popupcalculator:id/txtCalc";

    private static AccessibilityService mAccessibilityService;

    private final Context mContext;

    public CalculatorDemo(Context context) {
        mContext = context;
    }

    public void run() {
        if (mAccessibilityService == null) {
            Toast.makeText(mContext, "Accessibility service unavailable", Toast.LENGTH_SHORT).show();
            return;
        }

        // Start calculator app
        mContext.startActivity(new Intent().setClassName(PACKAGE, CLASS));
        sleep(2000);

        // Press button "1"
        findNodeInCurrentWindow(ID_BUTTON_01).performAction(AccessibilityNodeInfo.ACTION_CLICK);
        sleep(500);

        // Press button "+"
        findNodeInCurrentWindow(ID_BUTTON_ADD).performAction(AccessibilityNodeInfo.ACTION_CLICK);
        sleep(500);

        // Press button "2"
        findNodeInCurrentWindow(ID_BUTTON_02).performAction(AccessibilityNodeInfo.ACTION_CLICK);
        sleep(500);

        // Press button "="
        findNodeInCurrentWindow(ID_BUTTON_EQUAL).performAction(AccessibilityNodeInfo.ACTION_CLICK);
        sleep(500);

        // Read result
        String result = findNodeInCurrentWindow(ID_TEXT_DISPLAY).getText().toString();
        Toast.makeText(mContext, String.format("Result is %s", result), Toast.LENGTH_SHORT).show();

        // Go back
        mAccessibilityService.performGlobalAction(AccessibilityService.GLOBAL_ACTION_BACK);
    }

    public static void initialize(@NonNull AccessibilityService accessibilityService) {
        if (mAccessibilityService == null) {
            mAccessibilityService = accessibilityService;
        }
    }

    private static AccessibilityNodeInfo findNodeInCurrentWindow(String id) {
        AccessibilityNodeInfo window = mAccessibilityService.getRootInActiveWindow();
        return window.findAccessibilityNodeInfosByViewId(id).get(0);
    }

    private static void sleep(int milliseconds) {
        try {
            Thread.sleep(milliseconds);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
}

Enable accessibility service

After installing the CalculatorDemo APK, the service must be enabled in the Android settings:

1. Open Android settings
2. Go to «Accessibility»
3. In the «Services» section tap the service (e.g. «CalculatorDemo»)
4. Enable the service

Now the CalculatorDemo APK is ready to be run.