Skip to content

Instantly share code, notes, and snippets.

@gabriel-f-santos
Last active September 12, 2023 22:55
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save gabriel-f-santos/80b0aa87cde267626b5af9aa34d33f1a to your computer and use it in GitHub Desktop.
Save gabriel-f-santos/80b0aa87cde267626b5af9aa34d33f1a to your computer and use it in GitHub Desktop.
Logs elastic apm lambda extension cloudwatch
| INIT_START Runtime Version: python:3.9.v30 Runtime Version ARN: arn:aws:lambda:::runtime:86d4ce088432216337acec891c716c30002d0ed911f5a9574e36052e7527d6ab |
| {"log.level":"debug","@timestamp":"2023-09-12T17:29:55.189Z","log.origin":{"file.name":"extension/client.go","file.line":123},"message":"ExtensionID : ab65a2f9-3804-41ff-b5f0-e46f176d893f","ecs.version":"1.6.0"} |
| {"log.level":"debug","@timestamp":"2023-09-12T17:29:55.189Z","log.origin":{"file.name":"app/run.go","file.line":45},"message":"Register response: {\n\t\"functionName\": \"send-default\",\n\t\"functionVersion\": \"$LATEST\",\n\t\"handler\": \"handlers.send_notification\"\n}","ecs.version":"1.6.0"} |
| {"log.level":"info","@timestamp":"2023-09-12T17:29:55.190Z","log.origin":{"file.name":"apmproxy/receiver.go","file.line":57},"message":"Extension listening for apm data on :8200","ecs.version":"1.6.0"} |
| {"log.level":"info","@timestamp":"2023-09-12T17:29:55.194Z","log.origin":{"file.name":"logsapi/subscribe.go","file.line":73},"message":"Extension listening for Lambda Logs API events on 169.254.79.130:34301","ecs.version":"1.6.0"} |
| LOGS Name: apm-lambda-extension State: Subscribed Types: [Platform, Function] |
| {"log.level":"info","@timestamp":"2023-09-12T17:29:55.198Z","log.origin":{"file.name":"app/run.go","file.line":127},"message":"Waiting for next event...","ecs.version":"1.6.0"} |
| [DEBUG] 2023-09-12T17:29:55.561Z Inserting elasticapm log_record_factory into logging |
| [DEBUG] 2023-09-12T17:29:55.598Z Detected PID change from None to 17, starting threads |
| [DEBUG] 2023-09-12T17:29:55.598Z Starting transport thread |
| {"log.level":"debug","@timestamp":"2023-09-12T17:29:55.603Z","log.origin":{"file.name":"apmproxy/receiver.go","file.line":98},"message":"Handling APM server Info Request","ecs.version":"1.6.0"} |
| [DEBUG] 2023-09-12T17:29:55.609Z Skipping instrumentation of asyncpg. Module asyncpg.protocol.protocol not found |
| [DEBUG] 2023-09-12T17:29:55.610Z Fetched APM Server version 8.9.1 |
| [DEBUG] 2023-09-12T17:29:55.612Z Skipping instrumentation of aioredis. Module aioredis.pool not found |
| [DEBUG] 2023-09-12T17:29:55.614Z Skipping instrumentation of aiomysql. Module aiomysql.cursors not found |
| [DEBUG] 2023-09-12T17:29:55.615Z Skipping instrumentation of graphql. Module graphql.execution.executors.sync not found |
| [DEBUG] 2023-09-12T17:29:55.615Z Skipping instrumentation of graphql. Module graphql.execution.executors.gevent not found |
| [DEBUG] 2023-09-12T17:29:55.615Z Skipping instrumentation of graphql. Module graphql.execution.executors.asyncio not found |
| [DEBUG] 2023-09-12T17:29:55.615Z Skipping instrumentation of graphql. Module graphql.execution.executors.process not found |
| [DEBUG] 2023-09-12T17:29:55.616Z Skipping instrumentation of graphql. Module graphql.execution.executors.thread not found |
| [DEBUG] 2023-09-12T17:29:55.621Z Skipping instrumentation of sqlite. Module pysqlite2.dbapi2 not found |
| [DEBUG] 2023-09-12T17:29:55.621Z Instrumented sqlite, sqlite3.connect, sqlite3.dbapi2.connect |
| [DEBUG] 2023-09-12T17:29:55.623Z Skipping instrumentation of tornado_handle_request_exception. Module tornado.web not found |
| [DEBUG] 2023-09-12T17:29:55.624Z Skipping instrumentation of python_memcached. Module memcache not found |
| [DEBUG] 2023-09-12T17:29:55.722Z requests.packages.urllib3.connectionpool.HTTPConnectionPool.urlopen already instrumented, skipping |
| [DEBUG] 2023-09-12T17:29:55.730Z Skipping instrumentation of urllib3. Module botocore.vendored.requests.packages.urllib3.connectionpool not found |
| [DEBUG] 2023-09-12T17:29:55.730Z Instrumented urllib3, urllib3.connectionpool.HTTPConnectionPool.urlopen |
| [DEBUG] 2023-09-12T17:29:55.855Z Instrumented grpc_client_instrumentation, grpc.insecure_channel, grpc.secure_channel |
| [DEBUG] 2023-09-12T17:29:55.857Z Skipping instrumentation of starlette. Module starlette.middleware.errors not found |
| [DEBUG] 2023-09-12T17:29:55.858Z Skipping instrumentation of mysql_connector. Module mysql.connector not found |
| [DEBUG] 2023-09-12T17:29:55.859Z Instrumented asyncio_sleep, asyncio.tasks.sleep |
| [DEBUG] 2023-09-12T17:29:55.861Z Skipping instrumentation of elasticsearch_connection. Module elasticsearch.connection.http_urllib3 not found |
| [DEBUG] 2023-09-12T17:29:55.861Z Skipping instrumentation of elasticsearch_connection. Module elasticsearch.connection.http_requests not found |
| [DEBUG] 2023-09-12T17:29:55.862Z Skipping instrumentation of pymysql. Module pymysql not found |
| [DEBUG] 2023-09-12T17:29:55.864Z Skipping instrumentation of aiopg. Module aiopg.cursor not found |
| [DEBUG] 2023-09-12T17:29:55.864Z Skipping instrumentation of aiopg. Module aiopg.connection not found |
| [DEBUG] 2023-09-12T17:29:56.005Z Instrumented httplib2, httplib2.Http.request |
| [DEBUG] 2023-09-12T17:29:56.010Z Skipping instrumentation of azure. Module azure.core.pipeline._base not found |
| [DEBUG] 2023-09-12T17:29:56.010Z Skipping instrumentation of azure. Module azure.cosmosdb.table.common._http.httpclient not found |
| [DEBUG] 2023-09-12T17:29:56.012Z Skipping instrumentation of pyodbc. Module pyodbc not found |
| [DEBUG] 2023-09-12T17:29:56.013Z Instrumented requests, requests.sessions.Session.send |
| [DEBUG] 2023-09-12T17:29:56.014Z Skipping instrumentation of pylibmc. Module pylibmc not found |
| [DEBUG] 2023-09-12T17:29:56.016Z Skipping instrumentation of httpcore. Module httpcore._sync.connection not found |
| [DEBUG] 2023-09-12T17:29:56.016Z Instrumented grpc_server_instrumentation, grpc.server |
| [DEBUG] 2023-09-12T17:29:56.017Z Skipping instrumentation of pymemcache. Module pymemcache.client.base not found |
| [DEBUG] 2023-09-12T17:29:56.017Z Skipping instrumentation of pymemcache. Module pymemcache.client.hash not found |
| [DEBUG] 2023-09-12T17:29:56.018Z Skipping instrumentation of pymssql. Module pymssql not found |
| [DEBUG] 2023-09-12T17:29:56.019Z Skipping instrumentation of httpx. Module httpx not found |
| [DEBUG] 2023-09-12T17:29:56.021Z Skipping instrumentation of redis. Module redis.client not found |
| [DEBUG] 2023-09-12T17:29:56.022Z Skipping instrumentation of redis. Module redis.asyncio.client not found |
| [DEBUG] 2023-09-12T17:29:56.022Z Skipping instrumentation of aioredis. Module aioredis.commands.transaction not found |
| [DEBUG] 2023-09-12T17:29:56.083Z Instrumented psycopg2, psycopg2.extensions.register_type, psycopg2._json.register_json, psycopg2.extensions.quote_ident, psycopg2.extensions.encrypt_password |
| [DEBUG] 2023-09-12T17:29:56.085Z Skipping instrumentation of cassandra. Module cassandra.cluster not found |
| [DEBUG] 2023-09-12T17:29:56.218Z Instrumented jinja2, jinja2.Template.render |
| [DEBUG] 2023-09-12T17:29:56.220Z Skipping instrumentation of elasticsearch_connection. Module elasticsearch._async.transport not found |
| [DEBUG] 2023-09-12T17:29:56.222Z Skipping instrumentation of psycopg. Module psycopg not found |
| [DEBUG] 2023-09-12T17:29:56.226Z Skipping instrumentation of aiobotocore. Module aiobotocore.client not found |
| [DEBUG] 2023-09-12T17:29:56.227Z Skipping instrumentation of mysql. Module MySQLdb not found |
| [DEBUG] 2023-09-12T17:29:56.227Z Skipping instrumentation of redis. Module redis.connection not found |
| [DEBUG] 2023-09-12T17:29:56.229Z Instrumented urllib, urllib.request.AbstractHTTPHandler.do_open |
| [DEBUG] 2023-09-12T17:29:56.229Z Skipping instrumentation of elasticsearch_connection. Module elasticsearch_async.connection not found |
| [DEBUG] 2023-09-12T17:29:56.229Z Skipping instrumentation of elasticsearch_connection. Module elasticsearch._async.http_aiohttp not found |
| [DEBUG] 2023-09-12T17:29:56.230Z Skipping instrumentation of redis. Module redis.asyncio.client not found |
| [DEBUG] 2023-09-12T17:29:56.231Z Skipping instrumentation of httpx. Module httpx not found |
| [DEBUG] 2023-09-12T17:29:56.231Z Skipping instrumentation of tornado_request_execute. Module tornado.web not found |
| [DEBUG] 2023-09-12T17:29:56.231Z Skipping instrumentation of graphql. Module graphql.backend.core not found |
| [DEBUG] 2023-09-12T17:29:56.232Z Skipping instrumentation of graphql. Module graphql.backend.cache not found |
| [DEBUG] 2023-09-12T17:29:56.233Z Skipping instrumentation of aiohttp_client. Module aiohttp.client not found |
| [DEBUG] 2023-09-12T17:29:56.233Z Skipping instrumentation of tornado_render. Module tornado.web not found |
| [DEBUG] 2023-09-12T17:29:56.235Z Skipping instrumentation of kafka. Module kafka not found |
| [DEBUG] 2023-09-12T17:29:56.235Z Skipping instrumentation of aioredis. Module aioredis.connection not found |
| [DEBUG] 2023-09-12T17:29:56.235Z Skipping instrumentation of aioredis. Module aioredis.pool not found |
| [DEBUG] 2023-09-12T17:29:56.534Z Instrumented botocore, botocore.client.BaseClient._make_api_call |
| [DEBUG] 2023-09-12T17:29:56.536Z Skipping instrumentation of django_template_source. Module django.template.base not found |
| [DEBUG] 2023-09-12T17:29:56.539Z Skipping instrumentation of pymongo. Module pymongo.cursor not found |
| [DEBUG] 2023-09-12T17:29:56.540Z Skipping instrumentation of pymongo. Module pymongo.bulk not found |
| [DEBUG] 2023-09-12T17:29:56.541Z Skipping instrumentation of httpcore. Module httpcore._async.connection not found |
| [DEBUG] 2023-09-12T17:29:56.541Z Skipping instrumentation of pymongo. Module pymongo.collection not found |
| [DEBUG] 2023-09-12T17:29:56.541Z Skipping instrumentation of elasticsearch_connection. Module elasticsearch.transport not found |
| [DEBUG] 2023-09-12T17:29:56.542Z Skipping instrumentation of redis. Module redis.client not found |
| [DEBUG] 2023-09-12T17:29:56.542Z Instrumented grpc_async_server_instrumentation, grpc.aio.server |
| [DEBUG] 2023-09-12T17:29:56.542Z Instrumented psycopg2, psycopg2.connect |
| [DEBUG] 2023-09-12T17:29:56.542Z Skipping instrumentation of django_template. Module django.template not found |
| EXTENSION Name: apm-lambda-extension State: Ready Events: [INVOKE, SHUTDOWN] |
| ... |
| START RequestId: 6506fb66-7fc3-520b-be4e-a61cdbbb2ba5 Version: $LATEST |
| {"log.level":"debug","@timestamp":"2023-09-12T17:39:05.738Z","log.origin":{"file.name":"app/run.go","file.line":144},"message":"Received event.","ecs.version":"1.6.0"} |
| {"log.level":"debug","@timestamp":"2023-09-12T17:39:05.738Z","log.origin":{"file.name":"app/run.go","file.line":145},"message":"{\n\t\"timestamp\": \"2023-09-12T17:39:05.738139457Z\",\n\t\"eventType\": \"INVOKE\",\n\t\"deadlineMs\": 1694540385737,\n\t\"requestId\": \"6506fb66-7fc3-520b-be4e-a61cdbbb2ba5\",\n\t\"invokedFunctionArn\": \"arn:aws:lambda:::function:send-default\",\n\t\"tracing\": {\n\t\t\"type\": \"X-Amzn-Trace-Id\",\n\t\t\"value\": \"Root=1-6500a239-3ab2b86d2bf3a637500c0122;Parent=5c00738e78f88aed;Sampled=0;Lineage=6e04700a:0\"\n\t}\n}","ecs.version":"1.6.0"} |
| {"log.level":"debug","@timestamp":"2023-09-12T17:39:05.738Z","log.origin":{"file.name":"app/run.go","file.line":228},"message":"APM client has sent flush signal","ecs.version":"1.6.0"} |
| {"log.level":"debug","@timestamp":"2023-09-12T17:39:05.738Z","log.origin":{"file.name":"app/run.go","file.line":95},"message":"Waiting for background data send to end","ecs.version":"1.6.0"} |
| {"log.level":"debug","@timestamp":"2023-09-12T17:39:05.738Z","log.origin":{"file.name":"apmproxy/apmserver.go","file.line":58},"message":"Invocation context cancelled, not processing any more agent data","ecs.version":"1.6.0"} |
| {"log.level":"info","@timestamp":"2023-09-12T17:39:05.738Z","log.origin":{"file.name":"app/run.go","file.line":127},"message":"Waiting for next event...","ecs.version":"1.6.0"} |
| {"log.level":"debug","@timestamp":"2023-09-12T17:39:05.738Z","log.origin":{"file.name":"logsapi/event.go","file.line":137},"message":"Current invocation over. Interrupting logs processing goroutine","ecs.version":"1.6.0"} |
| [DEBUG] 2023-09-12T17:39:05.739Z 6506fb66-7fc3-520b-be4e-a61cdbbb2ba5 Sending partial transaction and early metadata to the lambda extension... |
| [DEBUG] 2023-09-12T17:39:05.740Z 6506fb66-7fc3-520b-be4e-a61cdbbb2ba5 http://127.0.0.1:8200 "POST /register/transaction HTTP/1.1" 200 0 |
| [DEBUG] 2023-09-12T17:39:05.741Z 6506fb66-7fc3-520b-be4e-a61cdbbb2ba5 Sent request, url=http://127.0.0.1:8200/register/transaction size=1.70kb status=200 |
| [DEBUG] 2023-09-12T17:39:05.741Z 6506fb66-7fc3-520b-be4e-a61cdbbb2ba5 Sending partial transaction and early metadata to the lambda extension... |
| [DEBUG] 2023-09-12T17:39:05.760Z 6506fb66-7fc3-520b-be4e-a61cdbbb2ba5 http://127.0.0.1:8200 "POST /register/transaction HTTP/1.1" 200 0 |
| [DEBUG] 2023-09-12T17:39:05.760Z 6506fb66-7fc3-520b-be4e-a61cdbbb2ba5 Sent request, url=http://127.0.0.1:8200/register/transaction size=1.70kb status=200 |
| [DEBUG] 2023-09-12T17:39:05.761Z 6506fb66-7fc3-520b-be4e-a61cdbbb2ba5 Received data: {'Records': [{'messageId': 'c4aa7555-aa57-4251-ac2f-8a0e31b76556', 'receiptHandle': 'AQEBzZJBaI1EHJ6ypANtouVnchBqsPrMONlmnDlBkTckZnfKCIelz5CI3pihytGBBrJGksrlz/sE/CJZGXwaGmgcIw4wOXgxcHdCFXf4RQmOCv0HSLI+nY7SU7kvttJKLsEE2zBLxPfNBPeBpiSV5UwjlDQ/4vT0iHPxWOBXbu9RHXeBDXHGgmpccJH92iBtUgVlhPTBQRLSkHL39pkz0k5i7BZpzipp9Gq855ZvB70mCAo+nmht8Z3xk/dKkoOiM2MrBdKiuWGb50XcwcJKKch44t8gknQSJlxej4CjfsPzlRDR3NfxcIW4Rhz36PKhWZGjugziY6WWm4icJDI3PxDuleCxZ9zU88mzJ5S3Sm0UtN6MTc8yWimiq5O9kxR/xrce72VLJ7+5GaBwQEjtUW35AhUEnKOZMxgBnIWPYXlpOEo=', 'body': '{"type": "email", "event_id": "b442a569-877b-49f5-be99-3d958358d9dd", "to": "email@email.com", "subject": "Some Subject", "sender": "sender@email.com", "from_sender": "custom_sender@email.com", "referrer": "test/test", "data": {"foo": "Test"}, "template": "test_template.html"}', 'attributes': {'ApproximateReceiveCount': '1', 'SentTimestamp': '1694540345650', 'SenderId': 'AIDAYADG7OK3AUM5TBLBM', 'ApproximateFirstReceiveTimestamp': '1694540345653'}, 'messageAttributes': {}, 'md5OfBody': '493d394c3f9dcd9af68e9107a1bbb655', 'eventSource': 'aws:sqs', 'eventSourceARN': 'arn:aws:sqs:::default-queue', 'awsRegion': 'us-east-1'}]} |
| [DEBUG] 2023-09-12T17:39:05.782Z 6506fb66-7fc3-520b-be4e-a61cdbbb2ba5 from_sender: custom_sender@email.com sender: sender@email.com |
| [DEBUG] 2023-09-12T17:39:05.782Z 6506fb66-7fc3-520b-be4e-a61cdbbb2ba5 Collecting metrics |
| [DEBUG] 2023-09-12T17:39:05.783Z 6506fb66-7fc3-520b-be4e-a61cdbbb2ba5 Flushing elasticapm data |
| [DEBUG] 2023-09-12T17:39:05.800Z 6506fb66-7fc3-520b-be4e-a61cdbbb2ba5 forced flush |
| {"log.level":"debug","@timestamp":"2023-09-12T17:39:05.801Z","log.origin":{"file.name":"apmproxy/receiver.go","file.line":118},"message":"Handling APM Data Intake","ecs.version":"1.6.0"} |
| [DEBUG] 2023-09-12T17:39:05.802Z 6506fb66-7fc3-520b-be4e-a61cdbbb2ba5 http://127.0.0.1:8200 "POST /intake/v2/events?flushed=true HTTP/1.1" 202 2 |
| [DEBUG] 2023-09-12T17:39:05.802Z 6506fb66-7fc3-520b-be4e-a61cdbbb2ba5 Sent request, url=http://127.0.0.1:8200/intake/v2/events?flushed=true size=2.02kb status=202 |
| [DEBUG] 2023-09-12T17:39:05.802Z 6506fb66-7fc3-520b-be4e-a61cdbbb2ba5 Flush complete |
| [DEBUG] 2023-09-12T17:39:05.802Z 6506fb66-7fc3-520b-be4e-a61cdbbb2ba5 Collecting metrics |
| [DEBUG] 2023-09-12T17:39:05.802Z 6506fb66-7fc3-520b-be4e-a61cdbbb2ba5 Flushing elasticapm data |
| [DEBUG] 2023-09-12T17:39:05.820Z 6506fb66-7fc3-520b-be4e-a61cdbbb2ba5 forced flush |
| {"log.level":"debug","@timestamp":"2023-09-12T17:39:05.821Z","log.origin":{"file.name":"apmproxy/receiver.go","file.line":118},"message":"Handling APM Data Intake","ecs.version":"1.6.0"} |
| [DEBUG] 2023-09-12T17:39:05.821Z 6506fb66-7fc3-520b-be4e-a61cdbbb2ba5 http://127.0.0.1:8200 "POST /intake/v2/events?flushed=true HTTP/1.1" 202 2 |
| [DEBUG] 2023-09-12T17:39:05.821Z 6506fb66-7fc3-520b-be4e-a61cdbbb2ba5 Sent request, url=http://127.0.0.1:8200/intake/v2/events?flushed=true size=0.00kb status=202 |
| [DEBUG] 2023-09-12T17:39:05.822Z 6506fb66-7fc3-520b-be4e-a61cdbbb2ba5 Flush complete |
| END RequestId: 6506fb66-7fc3-520b-be4e-a61cdbbb2ba5 |
| REPORT RequestId: 6506fb66-7fc3-520b-be4e-a61cdbbb2ba5 Duration: 102.40 ms Billed Duration: 103 ms Memory Size: 256 MB Max Memory Used: 177 MB |
| {"log.level":"debug","@timestamp":"2023-09-12T17:45:31.621Z","log.origin":{"file.name":"app/run.go","file.line":144},"message":"Received event.","ecs.version":"1.6.0"} |
| {"log.level":"debug","@timestamp":"2023-09-12T17:45:31.621Z","log.origin":{"file.name":"app/run.go","file.line":145},"message":"{\n\t\"timestamp\": \"2023-09-12T17:45:31.621811246Z\",\n\t\"eventType\": \"SHUTDOWN\",\n\t\"shutdownReason\": \"spindown\",\n\t\"deadlineMs\": 1694540733611,\n\t\"requestId\": \"\",\n\t\"invokedFunctionArn\": \"\",\n\t\"tracing\": {\n\t\t\"type\": \"\",\n\t\t\"value\": \"\"\n\t}\n}","ecs.version":"1.6.0"} |
| {"log.level":"debug","@timestamp":"2023-09-12T17:45:31.621Z","log.origin":{"file.name":"app/run.go","file.line":164},"message":"Received shutdown event with reason spindown","ecs.version":"1.6.0"} |
| {"log.level":"debug","@timestamp":"2023-09-12T17:45:31.621Z","log.origin":{"file.name":"app/run.go","file.line":228},"message":"APM client has sent flush signal","ecs.version":"1.6.0"} |
| {"log.level":"debug","@timestamp":"2023-09-12T17:45:31.640Z","log.origin":{"file.name":"app/run.go","file.line":95},"message":"Waiting for background data send to end","ecs.version":"1.6.0"} |
| {"log.level":"debug","@timestamp":"2023-09-12T17:45:31.640Z","log.origin":{"file.name":"logsapi/event.go","file.line":137},"message":"Current invocation over. Interrupting logs processing goroutine","ecs.version":"1.6.0"} |
| {"log.level":"warn","@timestamp":"2023-09-12T17:45:31.641Z","log.origin":{"file.name":"apmproxy/apmserver.go","file.line":344},"message":"Dropping agent data due to error: batch is full","ecs.version":"1.6.0"} |
| {"log.level":"debug","@timestamp":"2023-09-12T17:45:31.702Z","log.origin":{"file.name":"apmproxy/apmserver.go","file.line":169},"message":"Sending data chunk to APM server","ecs.version":"1.6.0"} |
| {"log.level":"debug","@timestamp":"2023-09-12T17:45:31.735Z","log.origin":{"file.name":"apmproxy/apmserver.go","file.line":58},"message":"Invocation context cancelled, not processing any more agent data","ecs.version":"1.6.0"} |
| {"log.level":"info","@timestamp":"2023-09-12T17:45:31.735Z","log.origin":{"file.name":"app/run.go","file.line":98},"message":"Exiting due to shutdown event with reason spindown","ecs.version":"1.6.0"} |
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment