[IT] Deploying a Personal Django App to Heroku for Production (Part 2 (AWS Part 2): Upload-only IAM / Connectivity Test / Heroku Integration)

Hello everyone.

Continuing from last time,
we will build an environment for deploying a personally developed Django app to Heroku for production.

Series Navigation (5-part series + Prologue)

• Prologue: Design philosophy, overview, prerequisites, naming, and operational policy

• Part 1 (AWS Part 1): S3 Creation / Lifecycle / "Gleaning" Lambda

• Part 2 (AWS Part 2) (This article): Upload-only IAM / Connectivity Test / Heroku Integration (This article)

• Part 3 (Heroku Part 1): Config Vars / Release / Minimal Static File Operation

• Part 4 (Heroku Part 2): Daily Automated Backup (Scheduler + `pg_dump` + S3)

• Part 5 (Operations): Recovery Test & Alerts

* Please note that the number of installments and content may change.

Step 1: Create an upload-only IAM user (No deletion / Minimum privileges)

1-1. Design Policy (Reconfirmation)

• Upload handler (Heroku side): Put only. No Delete allowed.

• Deletion handler (Lambda): Created in Part 1. Delete permission is granted only to the Lambda role.

• This blocks the risk of accidental deletion at the IAM level.

1-2. Actions to grant (Minimal)

Uploads (especially multipart uploads) require several auxiliary actions in addition to Put.

• `s3:PutObject` (Required)

• `s3:AbortMultipartUpload` (Permission to abort MPU in progress)

• `s3:ListBucketMultipartUploads` (Reference to "list of ongoing MPUs" on the bucket side)

• `s3:ListMultipartUploadParts` (refer to the “MPU part list” on the object side)

• `s3:ListBucket` (limited to prefix used to check recent uploads / optional)
  ※ Get is not required (reading is not necessary, so do not include it)

1-3. Policy (limited to bucket + specific prefix)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListBucketLimitedToPrefix",
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:ListBucketMultipartUploads"],
      "Resource": "arn:aws:s3:::<YOUR_S3_BUCKET>",
      "Condition": {
        "StringLike": { "s3:prefix": ["<YOUR_PREFIX>/*"] }
      }
    },
    {
      "Sid": "PutAndMultipartOnlyWithinPrefix",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Resource": "arn:aws:s3:::<YOUR_S3_BUCKET>/<YOUR_PREFIX>/*"
    }
  ]
}

1-4. Creation Procedure (Console users)

1. IAM > Policies > Create → Paste the above JSON & save (e.g., `S3PutOnly-<APP_NAME>`).

2. IAM > Users > Create user

   • User name: `heroku-uploader-<APP_NAME>`

   • Access type: Access key - Programmatic access

   • Next → In “Attach policies directly”, select the policy from 1 → Create.

3. On the user details screen, Create access key → Store the CSV in a safe place.

1-5. Creation Procedure (CLI users | Optional)

# 事前変数
BUCKET="<YOUR_S3_BUCKET>"
PREFIX="<YOUR_PREFIX>"
APP="yourapp"

# ポリシーJSONを policy.json として保存後:
aws iam create-policy --policy-name S3PutOnly-$APP --policy-document file://policy.json

aws iam create-user --user-name heroku-uploader-$APP
aws iam attach-user-policy \
  --user-name heroku-uploader-$APP \
  --policy-arn arn:aws:iam::<ACCOUNT_ID>:policy/S3PutOnly-$APP

aws iam create-access-key --user-name heroku-uploader-$APP
# => AccessKeyId / SecretAccessKey を控える（CSV推奨）

Step 2: Register environment variables to Heroku (Config Vars)

Go to Heroku Dashboard → Settings → Reveal Config Vars and set the following.

Step 3: Connectivity Test (Single Upload → Multipart Upload)

Before running the backup script from Part 1 (`scripts/s3_backup.sh`), first perform a minimal check.

3-1. Single Upload (Verify `PutObject` only)

Perform a one-line test using a Heroku one-off dyno:

heroku run --app <yourapp> -- python - <<'PY'
import os, boto3, datetime
s3 = boto3.client("s3", region_name=os.getenv("AWS_DEFAULT_REGION"))
bucket = os.getenv("S3_BUCKET"); prefix = os.getenv("S3_PREFIX")
key = f"{prefix}/probes/putobject-{datetime.datetime.utcnow().isoformat()}.txt"
s3.put_object(Bucket=bucket, Key=key, Body=b"hello", ServerSideEncryption=os.getenv("S3_SSE","AES256"))
print("OK:", key)
exit()
PY

• Success → `.../probes/putobject-*.txt` is created in S3.

• Failure → Caused by either environment variables, permissions, or region.

3-2. Multipart Upload (Verify MPU-related permissions)

If over 5MB, boto3's `S3Transfer` will use MPU. Test to force it:

heroku run --app <yourapp> -- python - <<'PY'
import os, io, boto3, datetime
from boto3.s3.transfer import TransferConfig
size = 8 * 1024 * 1024  # 8MB
body = io.BytesIO(b"x" * size)
cfg = TransferConfig(multipart_threshold=5*1024*1024, multipart_chunksize=5*1024*1024)
s3 = boto3.client("s3", region_name=os.getenv("AWS_DEFAULT_REGION"))
bucket = os.getenv("S3_BUCKET"); prefix=os.getenv("S3_PREFIX")
key = f"{prefix}/probes/mpu-{datetime.datetime.utcnow().isoformat()}.bin"
s3.upload_fileobj(body, bucket, key, ExtraArgs={"ServerSideEncryption": os.getenv("S3_SSE","AES256")}, Config=cfg)
print("OK (multipart):", key)
exit()
PY

• Success → `PutObject` / `ListMultipartUploadParts` / `ListBucketMultipartUploads` are functional.

• Even if interrupted, the lifecycle policy 'Abort incomplete MPU after 7 days' will clean it up.

Step 4: 'Pre-production' execution of the backup script

4-1. Create file: `scripts/s3_backup.sh`

# scripts/s3_backup.sh
#!/usr/bin/env bash
set -euo pipefail

export PATH="/app/.heroku/python/bin:/app/.apt/usr/bin:$PATH"
for d in /app/.apt/usr/lib/postgresql/*/bin; do
  [ -d "$d" ] && export PATH="$d:$PATH"
done


APP_NAME="mailsession"                      # ログ表示用
BUCKET="${S3_BUCKET:?S3_BUCKET missing}"
PREFIX="${S3_PREFIX:-heroku/${APP_NAME}}"
SSE="${S3_SSE:-AES256}"
KMS_KEY_ID="${S3_KMS_KEY_ID:-}"

STAMP="$(date -u +%Y%m%d-%H%M%S)"
OUT="/tmp/${APP_NAME}-${STAMP}.dump"

export PGSSLMODE=require  # Heroku PG は TLS 前提

echo "[backup] Start pg_dump to ${OUT}"
pg_dump -Fc --no-acl --no-owner "$DATABASE_URL" -f "$OUT"

S3_URI="s3://${BUCKET}/${PREFIX}/daily-${APP_NAME}-${STAMP}.dump"
echo "[backup] Upload to ${S3_URI}"

if [[ "$SSE" == "aws:kms" && -n "$KMS_KEY_ID" ]]; then
  aws s3 cp "$OUT" "$S3_URI" --sse aws:kms --sse-kms-key-id "$KMS_KEY_ID"
else
  aws s3 cp "$OUT" "$S3_URI" --sse AES256
fi

rm -f "$OUT"
echo "[backup] Done."

4-2. Add to `requirements.txt`

boto3

4-3. Execute via one-off dyno

# まず push（または GitHub 連携で自動デプロイ後）
git add scripts/s3_backup.sh requirements.txt
git commit -m "Add pg_dump_to_s3 script"
git push heroku main

# 実行
heroku run --app <yourapp> -- python scripts/s3_backup.sh

4-4. Main causes when things don't go well

• `Unable to locate credentials` → `AWS_*` is not set or misspelled.

• `AccessDenied` → The scope (bucket/prefix) of the IAM policy is incorrect.

• `Invalid SSE` → `S3_SSE` is set to something other than `AES256` (this series assumes no KMS).

• `pg_dump` failure → Reachability of `DATABASE_URL` / Postgres add-on.

• Incorrect S3 key location → Missing trailing slash in `S3_PREFIX`, etc. (Expected format is `.../<APP_NAME>/daily-...`).

Step 5: Key Rotation & Operational Tips

5-1. Rotation

1. Issue a new access key (IAM → Select User → Create access key).

2. Update Heroku Config Vars with the new values (you may keep the old values for now).

3. Verify connectivity via One-off (re-run steps 1 & 2 from Step 3).

4. If there are no issues, deactivate and delete the old access key.

5-2. Tips for Accident Prevention

• Separation of privileges: Delete is only for Lambda (the premise of this article).

• Fixing the Prefix: Use something like `S3_PREFIX=heroku/<APP_NAME>` to isolate per app.

• Do not grant unnecessary Get permissions: Do not assign read permissions.

• Check logs: Verify upload tests and production execution via CloudWatch (Lambda) and the S3 object list.

• Visualize Heroku variables: If you have a `/admin/` page, it is also effective to display the "current S3 destination" in read-only mode on the admin page.

Frequently Asked Questions (Q&A)

Q. I want to set an “IP restriction” from Heroku to AWS.
A. Heroku's source IP is not stable. The `aws:SourceIp` condition in IAM policies is difficult in practice. The practical solution is to minimize privileges + limit prefixes + rotate short-term keys to mitigate risk.

Q. I want to install `awscli` and use `aws s3 cp`.
A. Installing `awscli` via apt on Heroku is prone to failure and unnecessary. Let's stick to boto3.

Q. I want to use KMS (SSE-KMS).
A. This is outside the scope of this series (future installment). If you use KMS, you need to design permissions and key policies for `SSEKMSKeyId` and `kms:Encrypt`.

Q. What about “daily backups” with Heroku Scheduler?
A. I will explain this in detail in Part 4. For now, the goal is to ensure connectivity via One-off.

Deliverables for this installment (Template summary)

• IAM Policy: `S3PutOnly-<APP_NAME>` (JSON above)

• IAM User: `heroku-uploader-<APP_NAME>` (Access key issuance)

• Heroku Config Vars: `AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_DEFAULT_REGION / S3_BUCKET / S3_PREFIX / S3_SSE`

• Connectivity Test: Single-shot (`put_object`) + MPU (`upload_fileobj` + `TransferConfig`)

Next installment preview (Part 3: Heroku Part 1)

• Re-checking the minimum configuration for Config Vars organization / Release / Static files (Whitenoise) for Heroku apps

• Pitfalls for production with `DEBUG=false` / `ALLOWED_HOSTS` / `CSRF_TRUSTED_ORIGINS`

• The administrator's initial `migrate` / `createsuperuser` in the "correct flag order"

See you
next time!

Previous article

Past articles

• #AWS

• #AmazonSES

• #CloudWatch

• #SNS

• #IAM

• #InfrastructureConstruction

• #CloudServices

• #EmailDelivery

• #EmailAuthentication

• #SPF

• #DKIM

• #DMARC

• #NotificationSystem

• #PersonalDevelopment

• #Startup

• #DevelopmentMemo

• #EngineersLearning

• #ImplementationRecord

• #TwoFactorAuthentication

• #Security

• #UserAuthentication

• #LoginFunction

• #Lambda

• #Django

• #Heroku

• #AWS

• #S3

• #Backup

• #EventBridge