Upload
ron-munitz
View
145
Download
3
Tags:
Embed Size (px)
Citation preview
PSCG
NDK Primer
Ron MunitzFounder & CEO - The PSCG
[email protected]@android-x86.orghttps://github.com/ronubo/
AnDevConBostonMay 2014
@ronubo
This work is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/4.0/
© Copyright Ron Munitz 2014
PSCG
About://Ron Munitz
● Distributed Fault Tolerant Avionic Systems○ Linux, VxWorks, very esoteric libraries, 0’s and 1’s
● Highly distributed video routers○ Linux
● Real Time, Embedded, Server bringups○ Linux, Android , VxWorks, Windows, devices, BSPs, DSPs,...
● Distributed Android○ Rdroid? Cloudroid? Too busy working to get over the legal naming, so
no name is officially claimed for my open source
● Enterprise Mobility, Enterprise Security, Privacy○ Security hardening, reverse engineering, and lots of good stuff ;-)○ Founded Nubo, the first remote Android workspace for the enterprise.○ New venture… Coming up!
@ronubo
PSCG
About://Ron Munitz
● What currently keeps me busy:○ Running the PSCG, an Embedded/Android consulting and Training
firm○ Teaching on behalf of the PSCG and New Circle○ Preparing the next generation of competent computer engineering
experts at Afeka, Tel-Aviv College of Engineering■ Teaching Android Internals, Android, OS, Cyber Security,...■ Leading the Cyber Security track ■ Advising final projects
○ Amazing present, endless opportunities. (Wish flying took less time)
● What will keep busy in the next month:○ All of the above○ The World Cup
■ Open to receive invitations to Brazil (and Latin America)!
@ronubo
PSCG
Agenda
● Part I: Java○ Java Native Interface (JNI) - Theory○ Hello World Tutorial - Practice
● Part II: JNI in the AOSP● Part III: Native Android Apps
IntroductionThe JNI is a native programming interface. It allows Java code that runs inside a Java Virtual Machine (VM) to interoperate with applications and libraries written in other programming languages, such as C, C++, and assembly.
All JVM implementations need to implement the JNI specification to allow compatibility with native code libraries.
JNI to Java ⇔ __asm to C
Motivation● The standard Java class library does not support the
platform-dependent features needed by the application.○ For example NEON, SSSE3, ...
● You already have a library written in another language, and wish to make it accessible to Java code through the JNI.
● You want to implement a small portion of time-critical code in a lower-level language such as assembly.
Mobile Motivation● Reduce CPU cycles (and battery drain rate!) on
intensive computations
● Add designated features for simple add-on hardware without having to support an entire ecosystem for that
● Media processing in Android: All is being done natively this way or another
● Porting legacy code
● Protecting your logic and algorithms from rev-eng
JNI Pitfalls● Subtle errors in the use of JNI can destabilize the entire JVM in
ways that are very difficult to reproduce and debug.
● An application that relies on JNI loses the platform portability Java offers ○ A partial workaround is to write a separate implementation of JNI code
for each platform and have Java detect the operating system and load the correct one at runtime.
○ How many different platforms are there? Is it really worth it?
● The JNI framework does not provide any automatic garbage collection for non-JVM memory resources allocated by code executing on the native side. Consequently, native side code (such as assembly language) must assume the responsibility for explicitly releasing any such memory resources that it itself acquires.
JNI environment pointers
The JNI interface is organized like a C++ virtual function table. It enables a VM to provide multiple versions of JNI function tables. The JNI interface pointer is only valid in the current thread. Native methods receive the JNI interface pointer as an argument. The VM is guaranteed to pass the same interface pointer to a native method when it makes multiple calls to the native method from the same Java thread.
Loading and linking native methodsNative methods are loaded with the System.loadLibrary() method. In the following example, the class initialization method loads a platform-specific native library in which the native method f is defined:
class Cls { native boolean f(int i, String s);
static { System.loadLibrary(“pkg_Cls”); } } The library pkg_Cls is platform specific. On a Unix derived system it will be named libpkg_Cls.so and on windows system it will be named pkg_Cls.dll.
Loading and linking native methods(cont.)
Dynamic linkers resolve entries based on their names. A native method name is concatenated from the following components:
● the prefix Java_● a mangled fully-qualified class name● an underscore (“_”) separator● a mangled method name
The signature of function f above is, as created by javah:
JNIEXPORT jboolean JNICALL Java_Cls_f(JNIEnv *, jobject, jint, jstring);JNIEnv * - pointer to the JNI interface
jobject - a pointer to the calling java class for a static method or a pointer to the calling java object for non-static method.
Using javah is “nice”. onLoad()-ing is nicer
● The previous slide is close to perfection.● But it is lightyears away.● In fact, there is no real need to use javah,
and when you are expereinced enough with the NDK, you will most likely never use it again.
● We’ll talk about it more when we talk about JNI in the AOSP.
Java Functions Calling Conventions● Java calling conventions (a simplified blast from the “past”):
○ Primitive types - Call by Value○ Reference types (objects) - Call by Reference
● The same rules apply when calling a JNI function from Java○ Primitive types, such as integers, characters, and so on, are
copied between Java and native code (call by value)○ Arbitrary Java objects, on the other hand, are passed by
reference.
JNI types and data structuresJava Type Native Type Description
boolean jboolean unsigned 8 bits
byte jbyte signed 8 bits
char jchar unsigned 16 bits
short jshort signed 16 bits
int jint signed 32 bits
long jlong signed 64 bits
float jfloat 32 bits
double jdouble 64 bits
Referencing Java Objects● The VM must keep track of all objects that have been passed to
the native code, so that these objects are not freed by the garbage collector.
● The native code, in turn, must have a way to inform the VM that it no longer needs the objects.○ Otherwise ⇒ Memory Leak
● The JNI divides object references used by the native code into two categories: local and global references.
Global and Local References
● Local references are valid for the duration of a native method call, and are automatically freed after the native method returns.
● They are the default type of reference
● Global references remain valid until they are explicitly freed.
● They are created (natively) by calling NewGlobalRef()
Referencing java objects (Cont.)● Objects are passed to native methods as local references. ● All Java objects returned by JNI functions are local references. ● The JNI allows the programmer to create global references from
local references. JNI functions that expect Java objects accept both global and local references. A native method may return a local or global reference to the VM as its result.
● Since the VM needs a certain amount of space to keep track of a local reference, creating too many local references may cause the system to run out of memory.
● Local references are only valid in the thread in which they are created.
● Local references are automatically garbage collected once the function returns.
Native → Java : Invocation API
● Used to call java from c/c++ code. ● The JNI interface pointer (JNIEnv) is valid only in the
current thread. ● Should another thread need to access the Java VM, it
must first call AttachCurrentThread() to attach itself to the VM and obtain a JNI interface pointer.
● Once attached to the VM, a native thread works just like an ordinary Java thread running inside a native method.
● The native thread remains attached to the VM until it calls DetachCurrentThread() to detach itself.
Accessing fields and methods● The JNI allows native code to access the fields and call the
methods of Java objects. ● The JNI identifies methods and fields by their symbolic names and
type signatures. ● A two-step process factors out the cost of locating the field or
method from its name and signature. ○ Obtain method ID○ Invoke method by method ID
Accessing fields and methods - Example
● Assume a “Java” class cls with the following function definition:○ double f (int someInt, String someString);
● Then, f can be invoked from native code as follows:
First: A method ID is obtained and cached: jmethodID methodId =
env->GetMethodID(cls, “f”, “(ILjava/lang/String;)D”);
Then, the native code can then use the method ID repeatedly without the cost of method lookup, as follows:
jdouble result = env->CallDoubleMethod(obj, methodID,10, str);
Exception handlingThere are two ways to handle an exception in native code:
● The native method can choose to return immediately, causing the exception to be thrown in the Java code that initiated the native method call.
● The native code can clear the exception by calling ExceptionClear(), and then execute its own exception-handling code.
After an exception has been raised, the native code must first clear the exception before making other JNI calls.
Exception Handling - bionic gotchas
● When using the JNI with C++, we also use some sort of a C++ standard library.
● Standard libraries may vary between implementations, and indeed, Android’s default implementation does not support exception handling at all.
● This also applies to some other “expected” features such as STL and RTTI.
● The solution is to select among other C++ libraries such as gnustl, stlport, gabi++, or more (possibly your own). This is done in Application.mk and Android.mk○ LOCAL_CPP_FEATURES = exceptions ...○ APP_STL := ...
JVM Type SignaturesType Signature Java Type
Z boolean
B byte
C char
S short
I int
J long
F float
D double
L fully-qualified-class ; fully-qualified-class
[ type type[]
( arg-types ) ret-type method type
For example, the Java method:
long f(int n, String s, int[] arr);
has the following type signature:
(ILjava/lang/String;[I)J
Step 1: HelloWorld.javaclass HelloWorld{
private native void print();public static void main(String[] args){
new HelloWorld().print();}static {
System.loadLibrary("HelloWorld"); // Note: No “lib”, No “.so”.}
}
Compile: javac HelloWorld.java .Run: java HelloWorldFail: java.lang.UnsatisfiedLinkError: no HelloWorld in java.library.path . Not surprising…
Step 2: Auto Generate Headers~/edu/jni$ javah HelloWorld # Note: javah takes a class name:~/edu/jni$ cat HelloWorld.h /* DO NOT EDIT THIS FILE - it is machine generated */
#include <jni.h>
/* Header for class HelloWorld */
#ifndef _Included_HelloWorld
#define _Included_HelloWorld
#ifdef __cplusplus
extern "C" {
#endif
/* Class: HelloWorld Method: print Signature: ()V */
JNIEXPORT void JNICALL Java_HelloWorld_print (JNIEnv *, jobject); // “Java” - because I didn’t use a package.
#ifdef __cplusplus
}
#endif
#endif
Step 3: HelloWorld.c #include <stdio.h> #include "HelloWorld.h" JNIEXPORT void JNICALL Java_HelloWorld_print (JNIEnv *env, jobject obj) { printf("Hello World!\n"); }
Compile: gcc -shared -fPIC -I<YourJvmIncludePaths> HelloWorld.c -o libHelloWorld.soRun: In a couple of slides.Fail: Easily. GCC flags are subtle.
Step 3.5: Build the Shared Library
Let’s have a look at the following concrete example: gcc -shared -fPIC \ -I/usr/lib/jvm/jdk1.7.0/include/linux -I/usr/lib/jvm/jdk1.7.0/include \HelloWorld.c -o libHelloWorld.so
-shared: Create a shared object. (In Windows: a “DLL”)-fPIC: Position Independent Code - Always create Shared Object with that flag.-I<et. al>: Location of jni.h, and files included from there. -o libHelloWorld.so: Everything between “lib” and “.so” must match the name in loadLibrary().
Step 4: Run
java -Djava.library.path=. HelloWorld
Note: It is important to tell java where to look for your shared libraries. Otherwise you will see the same error listed in Step 1.
Hello World!
The NDK in a nutshell
● Available as a JNI “SDK” to Android application developers.○ In the platform itself there is no need for it, just as
there is no need for the Android SDK● Eases porting of existing Linux code to
Android○ As long as it is not heavily depended on glibc
features, X, QT/GTK etc.● Enables common code for multiple
architectures ○ On the nominal portable C/C++ code case ○ @see Application.mk
JNI in the Android platform
● @see frameworks/base/services/java/com/android/server/SystemServer.java
● This is where the Java Android OS starts.● And it is heavily relying on native functions
SystemServer.java revisitedpublic class SystemServer { private static final String TAG = "SystemServer"; //...
private static native void nativeInit(); /*Called to initialize native system services.*/ public static void main(String[] args) { // ...
// Set runtime libary property (Dalvik/ART/..) ...
// Set initial system time (to workaround some negative value bugs) ...
// Enable profiler snapshot if profiler is enabled ...
// Take care of HeapSize for the System Server which never dies ... // ...
System.loadLibrary("android_servers"); Slog.i(TAG, "Entered the Android system server!");
nativeInit(); /* Initializes native services - This is a JNI call to frameworks/base/services/jni/com_android_server_SystemServer.cpp …
See next slide*/
ServerThread thr = new ServerThread();
thr.initAndLoop(); // Note to self: so long init2()!!!
}
}
com_android_server_SystemServer.cpp
● Path: frameworks/base/services/jni/● namespace: Android● C, C++, java, JNI:
○ com/android/server/SystemServer.java○ com_android_server_SystemServer.cpp○ Note: Name convention should suffice.
However, when adding to the SystemServer, it is advised to also jniRegisterNativeMethods (@see libnativehelper/…)
● As per the code itself… static void android_server_SystemServer_nativeInit(JNIEnv* env, jobject clazz) { char propBuf[PROPERTY_VALUE_MAX]; property_get("system_init.startsensorservice", propBuf, "1"); if (strcmp(propBuf, "1") == 0) { SensorService::instantiate(); // Start the sensor service which is pure native } // @see frameworks/native/services/sensorservice}
onload.cpp
● jint JNI_OnLoad(JavaVM* vm, void* reserved) Called by default by the JVM upon System.loadLibrary()
● Loads the JNI Environment (uses JNI_VERSION_1_4)● Registers native servers by calling
register_android_server_<Name>(env); <Name> can be one of:PowerManagerService SerialServiceInputApplicationHandle InputWindowHandleInputManager LightsServiceAlarmManagerService UsbDeviceManagerUsbHostManager VibratorServiceSystemServer location_GpsLocationProvider location_FlpHardwareProvider connectivity_VpnAssetAtlasService ConsumerIrService
register_android_server_<Name> and jniRegisterNativeMethods
● As we saw, libandroid_servers’s JNI_OnLoad() registers each of the JNI servers.
● Each explicitly registers the exported JNI constructs, by providing their signatures to libnativehelper’s jniRegisterNativeMethods() method
● Which in turns calls JNI’s RegisterNatives() method.● This is also essential because the AOSP uses naming
conventions that do not start with the “Java_” prefix, and therefore cannot be invoked unless explicitly registered by RegisterNatives()
JNI_onLoad()
● The context of the VM is available only in the called thread.
● Which means that calling back Java is possible only if:○ It is a call back from a thread invoked by Java○ Somewhere a reference is stored globally for all/for
relevant threads● Do JVM caching on onLoad()● Free on onUnload()
Native project directory structure
● Source Control:○ ./ - Root. Home of manifest, ant/gradle, ...○ src - Java source files○ lib - Libaries. Including generated libraries and .so’s○ res - Resources○ assets - Assets (raw resources etc.)○ jni - Native code. Home of *.c*, *.h*, *.mk
■ Android.mk - Makefile■ Application.mk - Application definition (APP_*)
● Generated○ gen - generated java files○ obj - Native object files○ bin - Binaries
NDK App development with Android
1. Java code - as in Java’s JNI.2. Native code - as in Java’s JNI3. javah - provide class path (bin/…)4. Makefiles
a. Android.mk - Android makefile.i. e.g. include $(BUILD_SHARED_LIBRARY)
b. Application.mk - Application definitionsi. e.g. APP_ABI := all -
5. ndk-build - builds native code6. ant/gradle - builds APK and bundles .so’s in
lib
jni/Android.mk
LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)
LOCAL_MODULE := hello-jniLOCAL_SRC_FILES := hello-jni.c
include $(BUILD_SHARED_LIBRARY)
Application.mk
APP_ABI := x86 armeabi
obj/local/x86/libhello-jni.soobj/local/armeabi/libhello-jni.so
ndk-build
src/com/example/hellojni/HelloJni.java
package com.example.hellojni;
import android.app.Activity;
import android.widget.TextView;
import android.os.Bundle;
public class HelloJni extends Activity {
@Override
public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
TextView tv = new TextView(this);
tv.setText( stringFromJNI() ); setContentView(tv);
}
public native String stringFromJNI(); public native String unimplementedStringFromJNI(); // Catch: Call --> exception static { System.loadLibrary("hello-jni"); }}
The Native App Glue
● Enables writing fully native applications○ That is 0 java files
● Usually good for porting games● Or porting other heavy event machine logic● Requires implementation of your own Looper● Requires a manifest declaration:
<activity android:name="android.app.NativeActivity" android:label="@string/app_name" android:configChanges="orientation|keyboardHidden"> <!-- Tell NativeActivity the name of or .so --> <meta-data android:name="android.app.lib_name" android:value="native-activity" />
...
Publishing Applications
● Two approaches: Each has its pros and cons
● The first: Compile with all the libraries you can (APP_ABI := all)○ Easier to maintain.○ At the cost of larger (and sometimes huge) APK’s
● The second: Create version per architecture○ Create application with a slightly modified version
code at the MSB.○ This is the PlayStore way of identifying multiple
APKs for the same version
References● http://docs.oracle.com/javase/7/docs/technotes/guides/jni/spec/types.html● http://developer.android.com/tools/sdk/ndk/index.html● Introduction to Android Internals - Ron Munitz, Afeka