Short answers to the questions that come up most, with links to the guide that covers each topic in depth.
Getting started
What do I need before I can use staRburst?
- An AWS account, with credentials available to R (see credentials below).
-
Docker running on your machine. staRburst builds
the worker image locally with
docker buildxand pushes it to your ECR repository. You only pay this build once per environment (see the first run). - A one-time
starburst_setup(), which creates the S3 bucket, ECR repository, ECS cluster, IAM roles, and the default EC2 capacity provider.
The Getting Started guide walks through all three.
Which function should I use?
Start with starburst_map(). If you already have
future, future.apply, or furrr
code, set plan(starburst) and leave the rest unchanged. Use
starburst_session() for long jobs you want to close R and
come back to. The comparison table
has the full list.
EC2 or Fargate?
EC2 is the default and the right choice for most work. It’s cheaper
with Spot (also the default), lets you pick any instance type, and
supports warm pools. Fargate (launch_type = "FARGATE") is a
serverless option that needs no capacity provider, but it has cold-start
latency and is limited by your account’s Fargate vCPU quota. See Which
backend?.
Environment and data
Why is my first run so slow?
The first run builds a Docker image containing your R packages. That
takes roughly 5–10 minutes. The image is tagged with a hash of your
renv.lock and cached in ECR, so later runs with the same
lockfile reuse it. Changing your lockfile, for example by installing or
upgrading a package and snapshotting, triggers a rebuild.
Worker startup itself (launching instances and pulling the image) adds about 90 seconds per cold run. Running many jobs against one warm cluster spreads that cost out; see the Performance guide.
Which R packages are available on the workers?
The packages in your project’s renv.lock. If no lockfile
is found, staRburst creates one with renv::snapshot(). To
change what the workers have, update your lockfile. The next run detects
the new hash and rebuilds the image.
My R version has no public base image. What happens?
Public base images are published only for specific R releases
(currently 4.5.3, 4.6.1 and 4.6.2). On any other R version, staRburst
builds a private base image in your ECR instead, a one-time 3–5 minute
step per R version. Use staRburst 0.3.10 or later: earlier versions’
base-image template could silently produce images without
qs2, and workers built from them fail to read their
tasks.
What gets uploaded? Is my whole workspace sent?
No. For each task, staRburst uploads the function, that task’s input, and the globals the function actually references. Large datasets should go to S3 once, and each worker reads them from there. See Working with Data and the “don’t serialize huge objects to every worker” pitfall in the Performance guide.
Cost
How much will a job cost?
Estimate before you run. starburst_estimate() times a
small sample of your tasks locally and projects the cloud time and
cost:
starburst_estimate(inputs, slow_function, workers = 50)You can also set a ceiling so that over-budget jobs refuse to start:
starburst_config(max_hourly_cost = 10, cost_alert_threshold = 5)Reported costs are estimates: measured runtime multiplied by the current AWS price, looked up live and cached, with built-in rates as an offline fallback. They are not AWS billing figures.
What does staRburst cost when I’m not running anything?
Setup itself launches nothing billable. The EC2 capacity provider is created at zero instances. What persists between jobs:
-
ECR images. Storage for your worker images,
typically cents to a dollar a month. Pass
ecr_image_ttl_daystostarburst_setup()to expire old images automatically, or runstarburst_cleanup_ecr(). -
S3 objects. Job inputs and results are removed when
a job finishes, unless you set
starburst_config(auto_cleanup_s3 = FALSE). Detached sessions keep their results in S3 until you callsession$cleanup(force = TRUE). -
Warm pools. After a
plan(starburst)run or a detached session, the EC2 instances stay up forwarm_pool_timeoutseconds (default 10 minutes) past the last submission, so the next run starts quickly. They bill while they wait. The release is scheduled on AWS, so it happens even if R exits. Usewarm_pool_timeout = 0to release them as soon as the run ends.starburst_map()andstarburst_cluster()always release theirs when done. -
Detached-session workers. These keep running until
the session finishes, hits its
session_timeoutorabsolute_timeout, or you callsession$cleanup().
To confirm nothing is still running, check
starburst_list_sessions() and the EC2 and ECS consoles. The
Troubleshooting guide’s “High Costs /
Runaway Workers” section shows how to find and stop leftovers. Setting
an AWS Budgets alert is also a good idea.
Scale and limits
How many workers can I use?
Up to 500 per plan. staRburst refuses larger requests to protect you
from accidental runaway costs. On EC2, your account’s EC2 instance
limits also apply. On Fargate, your Fargate vCPU quota applies:
staRburst runs the job in waves if you ask for more than the quota
allows, and starburst_quota_status() and
starburst_request_quota_increase() help you raise it.
Why isn’t my job faster on the cloud?
Usually because each task is too small. Every task pays fixed overhead (serialization plus S3 round-trips), so thousands of sub-second tasks run slower than they would locally. Batch them into fewer, larger tasks, and use no more workers than you have tasks. The Workload Shapes guide covers this with measured numbers.
Accounts and operations
Which AWS credentials does it use?
Whatever the standard AWS credential chain finds, through the
paws SDK: environment variables, a named profile
(AWS_PROFILE), AWS SSO profiles, or an IAM role when
running on EC2 or ECS. Credentials never go in your code or in S3. The
workers use their own IAM role. See Security.
Which region does it run in?
The region you passed to starburst_setup() (default
us-east-1), stored in your staRburst config. Most functions
also take a region argument, but your bucket, repository,
and cluster live in the setup region.
Can I close R while a job is running?
Yes, with a detached session:
session <- starburst_session(workers = 20)
session$submit(quote(long_simulation()))
id <- session$session_id
# ...later, in a new R session:
session <- starburst_session_attach(id)
results <- session$collect(wait = TRUE)See the Detached Sessions guide.
Where are the worker logs?
In CloudWatch Logs. Read them from R with
starburst_logs(), or for a single task with
starburst_logs(task_id = ...). The Troubleshooting guide lists common error
patterns and what they mean.
