Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [outputes.0] HTTP Status=200 URI=/_bulk We are trying to get EKS logs to Graylog. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. [2022/03/24 04:19:54] [debug] [out coro] cb_destroy coro_id=4 fluentbit_output_proc_records_total. {"took":2033,"errors":true,"items":[{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"XeMnun8BI6SaBP9lLtm1","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. [warn]: temporarily failed to flush the buffer. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Please check my YAML for input and output plugin of the fluent bit. Fri, Mar 25 2022 3:08:42 pm | [2022/03/25 07:08:42] [debug] [out coro] cb_destroy coro_id=15 Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [outputes.0] task_id=8 assigned to thread #0 [2022/03/24 04:19:22] [debug] [out coro] cb_destroy coro_id=1 1 comment Closed . Fri, Mar 25 2022 3:08:27 pm | [2022/03/25 07:08:27] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=69479190 file has been deleted: /var/log/containers/hello-world-dsfcz_argo_main-13bb1b2c7e9d3e70003814aa3900bb9aef645cf5e3270e3ee4db0988240b9eff.log Setting Type doc in the es OUTPUT helped in my case. Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [input chunk] update output instances with new chunk size diff=695 [2022/03/24 04:20:00] [debug] [retry] re-using retry for task_id=2 attempts=4 Fri, Mar 25 2022 3:08:28 pm | [2022/03/25 07:08:28] [debug] [task] created task=0x7ff2f1839b20 id=6 OK Host 10.3.4.84 [2022/03/24 04:20:51] [error] [outputes.0] could not pack/validate JSON response Host {{ .Release.Name }}-elasticsearch-master, sassoftware/viya4-monitoring-kubernetes#431. If you see network-related messages, this may be an issue we already fixed in 1.8.15. Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [debug] [outputes.0] task_id=9 assigned to thread #1 Fri, Mar 25 2022 3:08:27 pm | [2022/03/25 07:08:27] [debug] [input chunk] update output instances with new chunk size diff=681 To Reproduce. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY a retry for websocket plugin has been triggered, with another handshake and data flush. Output always starts working after the restart. Fri, Mar 25 2022 3:08:28 pm | [2022/03/25 07:08:28] [debug] [retry] new retry created for task_id=6 attempts=1 I'm using fluentd in my kubernetes cluster to collect logs from the pods and send them to the elasticseach. Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY . no tls required for es. Fri, Mar 25 2022 3:08:33 pm | [2022/03/25 07:08:33] [debug] [outputes.0] task_id=11 assigned to thread #1 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"JeMmun8BI6SaBP9lh4vZ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Failed to create target, ioutil.ReadDir: readdirent: not a directory. Fri, Mar 25 2022 3:08:28 pm | [2022/03/25 07:08:28] [ warn] [engine] failed to flush chunk '1-1648192107.811048259.flb', retry in 11 seconds: task_id=6, input=tail.0 > output=es.0 (out_id=0) [2022/03/24 04:19:20] [debug] [input chunk] tail.0 is paused, cannot append records Fri, Mar 25 2022 3:08:44 pm | [2022/03/25 07:08:44] [ warn] [engine] failed to flush chunk '1-1648192124.833819.flb', retry in 10 seconds: task_id=17, input=tail.0 > output=es.0 (out_id=0) [2022/03/24 04:20:36] [error] [outputes.0] could not pack/validate JSON response [2022/03/24 04:20:36] [ warn] [engine] failed to flush chunk '1-1648095560.205735907.flb', retry in 13 seconds: task_id=0, input=tail.0 > output=es.0 (out_id=0) [2022/03/24 04:19:24] [ warn] [engine] failed to flush chunk '1-1648095560.254537600.flb', retry in 10 seconds: task_id=1, input=tail.0 > output=es.0 (out_id=0) "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"NuMmun8BI6SaBP9luqVq","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:28 pm | [2022/03/25 07:08:28] [debug] [out coro] cb_destroy coro_id=4 [2022/03/24 04:20:36] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000 Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [debug] [input chunk] update output instances with new chunk size diff=1182 Fri, Mar 25 2022 3:08:36 pm | [2022/03/25 07:08:36] [debug] [http_client] not using http_proxy for header The output plugins group events into chunks. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:42 pm | [2022/03/25 07:08:42] [debug] [outputes.0] task_id=16 assigned to thread #1 Fri, Mar 25 2022 3:08:48 pm | [2022/03/25 07:08:48] [debug] [out coro] cb_destroy coro_id=17 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"H-Mmun8BI6SaBP9lh4vZ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. (fbit, es) Make sure you're using either 1.9.1 or 1.8.15. [2022/03/24 04:19:24] [error] [outputes.0] could not pack/validate JSON response "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"YeMnun8BI6SaBP9lLtm1","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=69464185 removing file name /var/log/containers/hello-world-ctlp5_argo_main-276b9a264b409e931e48ca768d7a3f304b89c6673be86a8cc1e957538e9dd7ce.log "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"HuMoun8BI6SaBP9lIP7t","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. just like @lifeofmoo mentioned, initially everything went well in OpenSearch then the issue of "failed to flush chunk" came out. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Xyrion mentioned this issue. [2022/03/22 03:57:48] [ warn] [engine] failed to flush chunk '1-1647920969.178403746.flb', retry in 130 seconds: task_id=774, input=tail.0 > output=es.0 (out_id=0) [2022/03/24 04:19:22] [debug] [retry] new retry created for task_id=2 attempts=1 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"KeMoun8BI6SaBP9lIP7t","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamicall Fri, Mar 25 2022 3:08:46 pm | [2022/03/25 07:08:46] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 is now available Already on GitHub? [2022/03/24 04:19:21] [debug] [http_client] not using http_proxy for header Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [out coro] cb_destroy coro_id=22 hi @lecaros I think this was only [warn] message, I checked with es, I can search the right apps logs. FluentD or Collector pods are throwing errors similar to the following: 2022-01-28T05:59:48.087126221Z 2022-01-28 05:59:48 +0000 : [retry_default] failed to flush the buffer. [2022/03/24 04:20:20] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/hello-world-dsxks_argo_main-3bba9f6587b663e2ec8fde9f40424e43ccf8783cf5eafafc64486d405304f470.log, inode 35353618 Fri, Mar 25 2022 3:08:36 pm | [2022/03/25 07:08:36] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 has been assigned (recycled) Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [outputes.0] HTTP Status=200 URI=/_bulk [2022/03/24 04:21:20] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/fluent-bit-9hwpg_logging_fluent-bit-a7e85dd8e51db82db787e3386358a885ccff94c3411c8ba80a9a71598c01f387.log, inode 35353988 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"deMnun8BI6SaBP9l3vN-","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 has been assigned (recycled) But the situation is the same. Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Create a new Azure Storage account. Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Expected behavior Minimally that these messages do not tie-up fluent-bit's pipeline as retrying them will never succeed. Fri, Mar 25 2022 3:08:48 pm | [2022/03/25 07:08:48] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 has been assigned (recycled) Hi! [SERVICE] Flush 5 Daemon off [INPUT] Name cpu Tag fluent_bit [OUTPUT] Name forward Match * Host fd00:7fff:0:2:9c43:9bff:fe00:bb Port 24000. Are you still receiving some of the records on the ES side, or does it stopped receiving records altogether? I had similar issues with failed to flush chunk in fluent-bit logs, and eventually figured out that the index I was trying to send logs to already had a _type set to doc, while fluent-bit was trying to send with _type set to _doc (which is the default). [2022/03/24 04:19:34] [debug] [http_client] not using http_proxy for header Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. {"took":1923,"errors":true,"items":[{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"HeMoun8BI6SaBP9lIP7t","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Logs not being flushed after x amount of time. Bug Report. "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"kuMmun8BI6SaBP9l7LpS","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:27 pm | [2022/03/25 07:08:27] [debug] [out coro] cb_destroy coro_id=3 To easily locate the root cause and resolve this issue try AutoOps for Elasticsearch & OpenSearch. Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=3070975 removing file name /var/log/containers/hello-world-hxn5d_argo_wait-be32f13608de76af5bd4616dc826eebc306fb25eeb340049de8d3b8e5d40ba4b.log {"took":1935,"errors":true,"items":[{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"c-Mnun8BI6SaBP9l3vN-","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance].