Mobile Services | AWS Device Farm Flashcards

1
Q

What is AWS Device Farm?

General

AWS Device Farm | Mobile Services

A

AWS Device Farm allows developers to increase application quality, time to market, and customer satisfaction by testing and interacting with real Android and iOS devices in the AWS Cloud. Developers can upload their app and test scripts and run automated tests in parallel across 100s of real devices, getting results, screenshots, video, and performance data in minutes. They can also debug and reproduce customer issues by swiping, gesturing, and interacting with a device through their web browser.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

Who should use AWS Device Farm and why?

General

AWS Device Farm | Mobile Services

A

AWS Device Farm is designed for developers, QA teams, and customer support representatives who are building, testing, and supporting mobile apps to increase the quality of their apps. Application quality is increasingly important, and also getting complex due to the number of device models, variations in firmware and OS versions, carrier and manufacturer customizations, and dependencies on remote services and other apps. AWS Device Farm accelerates the development process by executing tests on multiple devices, giving developers, QA and support professionals the ability to perform automated tests and manual tasks like reproducing customer issues, exploratory testing of new functionality, and executing manual test plans. AWS Device Farm also offers significant savings by eliminating the need for internal device labs, lab managers, and automation infrastructure development.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

What types of apps does AWS Device Farm support?

General

AWS Device Farm | Mobile Services

A

AWS Device Farm supports native and hybrid Android, iOS, and web apps, and cross-platform apps including those created with PhoneGap, Titanium, Xamarin, Unity, and other frameworks.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

Does AWS Device Farm use simulators or emulators?

Setting Up Tests & Remote Access Sessions

AWS Device Farm | Mobile Services

A

AWS Device Farm tests are run on real, non-rooted devices. The devices are a mixture of OEM and carrier-branded devices.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

How do I get started with AWS Device Farm?

Setting Up Tests & Remote Access Sessions

AWS Device Farm | Mobile Services

A

Please see our getting started guide.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

Which browsers does the AWS Device Farm console support?

Setting Up Tests & Remote Access Sessions

AWS Device Farm | Mobile Services

A

AWS Device Farm works on Internet Explorer 9 or later and the latest versions of Chrome, Firefox, and Safari.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

Which browsers are supported for testing web applications?

Setting Up Tests & Remote Access Sessions

AWS Device Farm | Mobile Services

A

Your web applications will be tested in Chrome on Android and Safari on iOS.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

What is the maximum file size for apps and tests?

Setting Up Tests & Remote Access Sessions

AWS Device Farm | Mobile Services

A

AWS Device Farm supports files up to 4 GB.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

Do I need to instrument my app or supply source code?

Setting Up Tests & Remote Access Sessions

AWS Device Farm | Mobile Services

A

No instrumentation or source code is required to use the built-in tests. Android apps can be submitted as is. iOS apps should be built with “iOS Device” as the target instead of a simulator.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

Do you store my app, tests, and other files on your servers? For how long?

Setting Up Tests & Remote Access Sessions

AWS Device Farm | Mobile Services

A

Apps and files you upload are automatically removed after 30 days. Test results, screenshots, and videos are stored for 15 months. You can also choose to delete files and results at any time through the AWS Device Farm console or API.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

How do you clean up devices after my testing is completed?

Setting Up Tests & Remote Access Sessions

AWS Device Farm | Mobile Services

A

After test execution completes, we perform a series of cleanup tasks on each device, including uninstallation of your app. If we cannot verify uninstallation of your app or any of the other cleanup steps, the device will be removed and will no longer be available.

While we continue to add additional cleanup steps and improve the cleanup process, it is possible for data to persist between sessions in some cases, especially if you make use of the device system outside the context of your app. For this reason, and because AWS Device Farm captures video and logs of activity taking place during your use of each device, it is recommended that you avoid providing or entering sensitive information such as account info (E.g., Google Account, Apple ID), personal information, and other security-sensitive details during your automated test and remote access sessions.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

Do you modify my app?

Setting Up for Direct Device Access for Xcode and Android Studio

AWS Device Farm | Mobile Services

A

On iOS, we replace the embedded provisioning profile with a wildcard profile and resign the app. If you provide it, we will add auxiliary data to the application package before installation so the data will be present in your app’s sandbox. Resigning the iOS app results in the removal of certain entitlements. This includes App Group, Associated Domains, Game Center, HealthKit, HomeKit, Wireless Accessory Configuration, In-App Purchase, Inter-App Audio, Apple Pay, Push Notifications, and VPN Configuration & Control.

On Android, we resign the app. This may break functionality that depends on the app signature, such as the Google Maps Android API. It may also trigger anti-piracy and anti-tamper detection available from products such as DexGuard. For built-in tests, we may modify the manifest to include permissions required to capture and save screenshots.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

What is AWS Device Farm Direct Device Access?

Setting Up for Direct Device Access for Xcode and Android Studio

AWS Device Farm | Mobile Services

A

This feature allows developers to access their private fleet devices via their local host machines the same way they debug with real devices connected directly to the machines. On top of the ability for customers to perform the same tasks they do with real devices on Android Studio and Xcode today. Device Farm direct device access gives user the wide range selection of devices and OS versions. To learn more about private devices, please contact us.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

Does it support both Android and iOS?

Setting Up for Direct Device Access for Xcode and Android Studio

AWS Device Farm | Mobile Services

A

Yes.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

Do I need to additional software on my machine?

Setting Up for Direct Device Access for Xcode and Android Studio

AWS Device Farm | Mobile Services

A

Yes. You can down load the client software from one of the following locations.

Windows: https://s3-us-west-2.amazonaws.com/prod-us-west-2-system-resources/AWSDeviceFarmClient/AWS_Device_Farm.exe

MacOS: https://s3-us-west-2.amazonaws.com/prod-us-west-2-system-resources/AWSDeviceFarmClient/AWS_Device_Farm.dmg

For the full instructions for installation please read this documentation.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

Can I connect to multiple Android devices from my local machine?

Setting Up for Direct Device Access for Xcode and Android Studio

AWS Device Farm | Mobile Services

A

Yes.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

Does this work for public fleet of devices?

Setting Up for Direct Device Access for Xcode and Android Studio

AWS Device Farm | Mobile Services

A

No, at the moment, this feature is available for private fleets on AWS Device Farm. To get more information on private fleets: https://aws.amazon.com/device-farm/pricing/#privateDevices.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
18
Q

Can I debug and run on AWS Device Farm devices from Android Studio, Xcode, Chrome Developer Tools, Safari Web inspector?

Setting Up for Direct Device Access for Xcode and Android Studio

AWS Device Farm | Mobile Services

A

Yes.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
19
Q

What test frameworks are supported for Android devices ?

Setting Up for Direct Device Access for Xcode and Android Studio

AWS Device Farm | Mobile Services

A

Once you connect to Devices, you can run any framework of your choice.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
20
Q

Where can I find more information on this feature?

Selecting Devices

AWS Device Farm | Mobile Services

A

For the detail setup guide, please click here[insert production link]. To learn more about private devices, please contact us.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
21
Q

Which devices are available in AWS Device Farm? How do you select the devices in your fleet?

Selecting Devices

AWS Device Farm | Mobile Services

A

AWS Device Farm has a large (and growing) selection of Android, iOS, and Fire OS devices. We add popular new devices as they are released by manufacturers. We also add new devices as new OS versions are released. See the list of available devices.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
22
Q

Does AWS Device Farm have international devices from markets like Europe, China, and India?

Selecting Devices

AWS Device Farm | Mobile Services

A

We currently have international devices from India. We use market data and customer feedback to continuously update the fleet. If you would like to see a device that isn’t in our fleet, please let us know.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
23
Q

How do I select devices? Can I retest on the same device?

Selecting Devices

AWS Device Farm | Mobile Services

A

For Automated Testing, devices are selected through a collection called a device pool. Some curated device pools are provided automatically, but you can create your own pools, too. During execution, tests will be run against all devices in the specified pool that are compatible with your application and tests. For Remote Access, you select the desired device based on make, model, carrier variant, and operating system version. You can then optionally upload apps and other data as well as configure other device settings. Device Farm then locates an available device matching your request and displays the device’s display in your browser. You can then interact with the device and capture screenshots and video.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
24
Q

Are any apps pre-installed on AWS Device Farm test devices?

Selecting Devices

AWS Device Farm | Mobile Services

A

Yes, test devices will have a number of apps pre-installed by the device manufacturer or carrier.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
25
Q

Are devices able to communicate with other services or systems that are available on the Internet?

Selecting Devices

AWS Device Farm | Mobile Services

A

Yes. All devices have a WiFi connection with Internet access. If your systems are internal (that is, behind a corporate firewall), you can whitelist the IP range 54.244.50.32-54.244.50.63. All device traffic will come from those IPs.

26
Q

Can I test different carrier connections and conditions?

Selecting Devices

AWS Device Farm | Mobile Services

A

While you can’t test actual carrier connections, you can simulate connection types and conditions using the network shaping functionality. When scheduling a run, you can select a curated network profile like “3G” or “Lossy LTE,” or you can create your own, controlling parameters like throughput, jitter, and loss. All WiFi traffic from the device will be shaped and manipulated for the duration of your tests according to the profile you choose. You can also simulate dynamic environments by changing network parameters from your test scripts.

27
Q

Can I make phone calls or send SMS from the devices?

Selecting Devices

AWS Device Farm | Mobile Services

A

No, devices do not have carrier connections and cannot make phone calls or send SMS messages.

28
Q

Can I use the device camera?

Testing Your App

AWS Device Farm | Mobile Services

A

Yes, you can use the device cameras, both front- and rear-facing. Due to the way the devices are mounted, images and videos may look dark and blurry.

29
Q

I don’t have any automated test scripts yet. What do the built-in tests do?

Testing Your App

AWS Device Farm | Mobile Services

A

The built-in compatibility test suite allows you to install, uninstall, launch, and run Fuzz on the app.

30
Q

What does Fuzz do?

Testing Your App

AWS Device Farm | Mobile Services

A

Fuzz will perform fuzz testing on your UI immediately after launch. It streams random user input (touches, swipes, keyboard input) in a rapid fashion to your app. You can configure the number of events, the delay between events, and the seed used to randomize events. Using the same seed across test runs will result in the same sequence of events.

31
Q

I test using an automation framework. Which frameworks do you support?

Testing Your App

AWS Device Farm | Mobile Services

A

For testing iOS, Android, and FireOS apps, we currently support Appium Java JUnit, Appium Java TestNG, Appium Python, Calabash, Instrumentation (Including JUnit, Espresso, Robotium, and any instrumentation-based tests), UI Automation, UI Automator, and XCTest (Including XCUI and KIF). For more information and updated list, visit our documentation.

32
Q

Which test frameworks do you support for web applications?

Testing Your App

AWS Device Farm | Mobile Services

A

You can run tests written in Appium Java JUnit, Appium Java TestNG, or Appium Python.

33
Q

Can you add support for a modified framework or one I designed myself?

Testing Your App

AWS Device Farm | Mobile Services

A

We’re always evaluating frameworks to support. Please contact us.

34
Q

How does AWS Device Farm decide when to take a screenshot during a test?

Testing Your App

AWS Device Farm | Mobile Services

A

If you use one of the supported automation frameworks, you are in full control and can decide when to take screenshots. Those screenshots are included in your reports automatically.

35
Q

Android: Is Google Play Services available on your devices? Which version is installed?

Testing Your App

AWS Device Farm | Mobile Services

A

Yes, Google Play Services is installed on devices that support it. The services are updated as new versions become available.

36
Q

Android: Is there a default Google account on the devices?

Testing Your App

AWS Device Farm | Mobile Services

A

No, devices do not have an active Google account.

37
Q

Does AWS Device Farm support record and playback automation or do I have to write my scripts?

Testing Your App

AWS Device Farm | Mobile Services

A

AWS Device Farm supports frameworks like Espresso and Robotium that have record and playback scripting tools.

38
Q

iOS: Do I need to add your UDIDs to my provisioning profile?

Testing Your App

AWS Device Farm | Mobile Services

A

No, AWS Device Farm will automatically replace a provisioning profile and resign your app so it can be deployed on our devices.

39
Q

iOS: My app does not contain debug symbols. Can I supply a dSYM file to AWS Device Farm?

Testing Your App

AWS Device Farm | Mobile Services

A

No, but you can download the logs and symbolicate the stack traces locally.

40
Q

Android: My app is obfuscated. Can I still test my app on AWS Device Farm?

Testing Your App

AWS Device Farm | Mobile Services

A

Yes, if you use ProGuard. If you use DexGuard with anti-piracy measures, we are unable to re-sign the app and run tests against it.

41
Q

My app serves ads. Will they be displayed on your devices? Will my ad provider flag this as abuse and ban my account?

Testing Your App

AWS Device Farm | Mobile Services

A

Although devices have access to the Internet, we make no guarantee that ads will be displayed. We recommend that you remove ads from the builds tested on AWS Device Farm.

42
Q

Can I access the machine hosting the device or access its shell as part of my tests? Can I reach the Internet from it?

Testing Your App

AWS Device Farm | Mobile Services

A

Yes. If you’re using a client-server framework like Calabash, Appium, or UI Automation, you can access the Internet and execute limited shell commands from the host.

43
Q

I’d like to supply media or other data for my app to consume. How do I do that?

Testing Your App

AWS Device Farm | Mobile Services

A

You can provide a .zip archive up to 4 GB in size. On Android, it will be extracted to the root of external memory; on iOS, to your app’s sandbox. For Android expansion files (OBB), we will automatically place the file into the location appropriate to the OS version. For more information, see the Developer Guide.

44
Q

My app requires dependencies to test all functionality. Can I install other apps?

Testing Your App

AWS Device Farm | Mobile Services

A

Yes, you can select multiple apps and the order in which to install them. These dependent apps will be installed before your tests begin.

45
Q

Can I test upgrade flows for my app? How do I install an old version of my app?

Testing Your App

AWS Device Farm | Mobile Services

A

Yes, in order to test your upgrade flow, you can upload and install an old version of your app before the new version is installed and tested.

46
Q

My app makes use of location services. Can I specify the physical location of the device?

Testing Your App

AWS Device Farm | Mobile Services

A

Yes, you can supply latitude and longitude coordinates that will be used to override a device’s GPS.

47
Q

Can I run localization tests? How do I change the language of the device?

Testing Your App

AWS Device Farm | Mobile Services

A

Yes, you can provide a locale (for example, “en_US”) to override the default locale setting on a device.

48
Q

How long does it take before my test starts?

Testing Your App

AWS Device Farm | Mobile Services

A

Tests are immediately queued for execution and usually start within minutes. If one or more devices are not available, test execution for those devices will remain queued until they become available. Testing on the other devices in your test run will continue.

49
Q

What is the maximum test time allowed?

Testing Your App

AWS Device Farm | Mobile Services

A

The maximum time allowed is 60 minutes. If you need a longer timeout, please contact us.

50
Q

Does AWS Device Farm provide a way to run tests and get results through an API?

Reviewing Results

AWS Device Farm | Mobile Services

A

Yes. We have a plug-in for the Jenkins continuous integration environment and a Gradle plugin compatible with Android Studio. AWS Device Farm also provides programmatic support for all console features, including setting up a test and downloading test results through an API. For more information, see the AWS Device Farm API Reference. In addition to the API, you can access AWS Device Farm from the AWS SDKs.

51
Q

What’s in an AWS Device Farm test report?

Reviewing Results

AWS Device Farm | Mobile Services

A

AWS Device Farm test reports contain pass/fail information, crash reports, test logs, device logs, screenshots, videos, and performance data. Reports include both detailed per-device data and high-level results like the number of occurrences of a given error. Remote Access results contain logs and a video of the session.

52
Q

Which device logs are included in an AWS Device Farm report?

Reviewing Results

AWS Device Farm | Mobile Services

A

AWS Device Farm reports include complete logcat (Android) and device logs (iOS), as well as logs from the device host and specified test framework.

53
Q

My tests generate and save additional log files. Will I see them in my AWS Device Farm reports?

Pricing

AWS Device Farm | Mobile Services

A

If you write data to logcat (Android) or the device log (iOS), those log entries will be included in the report. AWS Device Farm does not collect any non-standard logs or other artifacts, although you may transfer files via your test script using the device’s or device host’s Internet connection.

54
Q

How much does AWS Device Farm cost?

Pricing

AWS Device Farm | Mobile Services

A

Pricing is based on device minutes, which are determined by the duration of tests on each selected device. AWS Device Farm comes with a free trial of 1000 device minutes.* After that, customers are charged $0.17 per device minute. As your testing needs grow, you can opt for an unmetered testing plan, which allows unlimited testing for a flat monthly fee of $250 per device.

55
Q

How does the free trial work?

Pricing

AWS Device Farm | Mobile Services

A

Your first 1000 device minutes are provided free of charge.* This is a one-time trial and does not renew. Once your trial allocation is depleted, you will be billed at the standard rate of $0.17 per device minute.

56
Q

What is a device minute?

Pricing

AWS Device Farm | Mobile Services

A

A device minute is the billing unit. Device minutes are a measurement of the time it takes (in minutes) to install, execute, and uninstall your app and tests on every device you have selected for your test run. The unit price is constant regardless of the device, test, or application type. Device minutes are only billed for tests that complete without any device or system errors. Similarly, for Remote Access sessions, device minutes are measured from the time it takes to prepare a device to your specification to completely removing any apps and data you placed on the device.

You can dynamically grow and shrink your usage according to your business needs without any upper limits or up-front commitments.

57
Q

What is the unmetered plan and how do device slots work?

Pricing

AWS Device Farm | Mobile Services

A

Unmetered plans allow unlimited testing and remote access starting at $250 per month. Unmetered pricing is based on the number of device slots you purchase for each usage type (i.e. automated test or remote access) and device family (i.e. Android or iOS) and are priced at $250 per slot per month. Device slots correspond to concurrency.

For instance, if you purchase ten automated test Android device slots and schedule a run on 100 Android devices, Device Farm will execute your tests on up to ten devices at a time until all tests are completed on your selected devices. Purchasing more slots would enable you to get your results faster. Regardless of how many tests or remote access sessions you have in a month, you are billed at the flat rate of $250 per device slot per month. You can cancel your subscription for one or more device slots at any time and the cancellation will take effect at your next renewal date (the day of the month that you purchased your first active device slot). To learn more, see our docs.

58
Q

What if my testing needs change and I need to add or remove device slots?

Pricing

AWS Device Farm | Mobile Services

A

You can add device slots at any time and they will be available to you immediately. You can also cancel your subscription for one or more device slots at any time and the cancellation will take effect at your next renewal date (the day of the month that you purchased your first active device slot).

59
Q

If I’m on an unmetered plan, can I still make use of metered billing?

Pricing

AWS Device Farm | Mobile Services

A

Yes. When creating a run, you can choose to make use of your unmetered device slots or use metered device minutes instead. Because concurrency is not limited on metered billing, this gives you the flexibility of running tests faster than would otherwise happen using your device slots.

60
Q

What is a private device?

Pricing

AWS Device Farm | Mobile Services

A

A private device is a physical instance of a phone or tablet that is exclusive to your account. Private devices can have custom, static configurations and run custom OS images. Each device is deployed on your behalf and removed at the end of your subscription.

61
Q

How do private device subscriptions work and how are they priced?

Pricing

AWS Device Farm | Mobile Services

A

Each private device under your account is considered a private device subscription. The monthly subscription price is tiered on the cost of the device and starts at $200/month. After the minimum subscription period, you can choose to cancel your subscription at any time. Please contact us for more information.