fire safety Archives

This post was written by Josh Weiner, Solar Expert Witness & Solar Engineering Expert and leads a discussion about safety in energy storage projects. Mr. Weiner has been at the forefront of the solar industry for over 20 years and is an industry leader on solar-plus-storage engineering & design. Josh’s expertise spans both in-front of and behind-the-meter initiatives including residential, commercial, utility, grid-scale, and ev charging solar and storage applications.

A recent Greentech Media article brought to light new details about a lithium battery fire at the Arizona Public Service (APS) McMicken Energy Storage facility that occurred in April.

According to GTM, the fire involved a thermal event affecting one battery rack but not a thermal runaway event affecting multiple battery racks. This is very good news, as we’ll explain below. The article also suggests that venting energy storage enclosures to release combustible gases may be a solution. We respectfully disagree.

A few months after the fire, we gave some initial suggestions on how project managers can design risk out of energy storage systems. This post presents our take on the facts as we now understand them.

We still don’t know the root cause of the fire. However, we know enough to conclude that more ventilation is not the best approach to battery fire prevention. We also know that storage projects need a failure plan, and they need to comply with higher standards.

Read on for our recommendations to help energy storage contractors prevent lithium battery fires.

No thermal runaway

After the Arizona fire, an investigation from APS and battery provider Fluence found that only one battery rack containing 14 modules had “melted.” Evidently the fire did not spread to adjacent racks, setting up a more hazardous thermal runaway scenario, which could have added to the fire’s propagation many times over.

This is encouraging. It’s not the failure of a single cell, but rather the propagation of that failed cell that causes all the damage we see in lithium fires. We should understand why the propagation stopped at the rack level. Here are a few possibilities.

(A) The spacing between racks in the system design was wide enough to stop the fire’s propagation in other racks.

(B) The original equipment manufacturer’s design of the battery pack itself helped prevent rack-to-rack propagation.

As the investigation proceeds, we hope to understand not only the root cause of the APS fire but also the design criteria that helped prevent rack-to-rack thermal runaway. APS is reporting investigation updates here. Independent research and third-party lab testing can also produce findings that improve design and engineering for battery safety.

Venting is not the answer

APS director of technology innovation and integration Scott Bordenkircher told GTM that the McMicken facility fire will prompt engineering and design changes, balancing fire suppression with the removal of explosive gases.

A better answer might be to make sure fire response professionals do not open containers designed to enclose and isolate what’s inside. Do we know enough today to arm firefighters with the correct training to protect themselves and suppress fire? A system designed to fully contain explosive gases may be part of the solution rather than the problem. While investigating ways to improve lithium battery safety, it’s also a good idea to explore best practices for first responders.

Root cause unknown

While it is encouraging that rack-to-rack propagation did not occur, the root cause of the APS fire is still unknown. A root cause analysis will help engineers modify future designs to improve lithium battery safety. Following the chain of events backwards to the point of origin (modules within the rack, cells within the module, and down to the cell level) can yield key insights.

If the root cause of the fire was truly “spontaneous,” which is a real possibility when large quantities of lithium cells are manufactured, no design or manufacturing changes can eliminate the possibility of another freak accident occurring. We may have to accept that spontaneous lithium failures are inherent in lithium technology and manufacturing processes. If this is the case, the best we can do is focus on controllable areas of fire suppression, isolation, and safety at the component- and system-level, rather than at the cell- or module-level.

With that in mind, what can energy storage companies do to eliminate or mitigate lithium battery fires? Here are two recommendations.

Plan for failure

In the event of a lithium battery fire, projects need clear and well-documented protocols to assist in fire suppression, cleanup, and investigation. These prevention and remediation plans ought to be provided as part of the project-specific safety plan or permitting process. This would ensure the information is provided to local authorities and site personnel. System design should also be informed by the possibility of system-level or component-level failures. Fire, building, chemical, and electrical safety codes and standards may be consulted and referenced.

For instance, in the APS fire, the bad rack was positioned in the middle of several batteries that maintained a 90% state of charge. As a result, the APS/Fluence team spent 9 weeks removing and de-energizing all of those batteries.

“There was absolutely no playbook,” Bordenkircher told GTM.

If this experience leads to the creation of a proactive project failure plan, that would be a positive outcome. It could help guide future safety code iterations and standards development.

In addition, it is interesting that APS used LG Chem batteries. According to SepiSolar research, LG Chem batteries have among the widest temperature range needed to initiate thermal runaway. LG Chem batteries also have a fire incident history that reportedly led the battery maker to shut down some of its own storage systems in South Korea.

Raise project standards

The risk of a lithium battery fire is lower in residential and commercial applications than in utility installations. The reason: such projects must comply with the UL 9540 and NFPA 855 safety standards. Utility projects, on the other hand, are basically self-regulating.

UL 9540 addresses construction, performance, and testing of energy storage systems, including how the system handles combustible concentrations and fire detection and suppression.

If we hold utility projects to higher safety standards, battery fire risks will go down.

Improve risk management

It’s more important than ever to understand and manage the risks associated with energy storage projects. That’s why SepiSolar is writing about the APS battery fire and why we will continue to write about it.

Our experience balancing cost, speed, and safety in energy storage projects contributed to the development of the new C&I Project Risk Management Guide. Download a free copy today.

Feature photo by Constante Ken Lim.

This post was written by Josh Weiner, Solar Expert Witness & Solar Engineering Expert. Mr. Weiner has been at the forefront of the solar energy industry for over 20 years and is an industry leader on solar-plus-storage engineering & design. Josh’s expertise spans both in-front of and behind-the-meter initiatives including residential, commercial, utility, grid-scale, and ev charging solar and storage applications.

Lithium battery safety is not rocket science. Manufacturers with a robust set of production data can show customers success rates for their batteries and the conditions that cause batteries to fail. The problem is that very little safety data is accessible to most buyers or the public.

Buyers will always have to decide for themselves how much risk they are willing to tolerate. Some source batteries from a selective group of original equipment manufacturers (OEMs) and pay a premium to avert risks associated with the lowest-priced batteries. But many buyers are operating in the dark, lacking the safety data they would need to make an informed decision.

Consequently, the energy storage industry in its brief history has already witnessed dangerous and damaging lithium battery safety incidents, including the April 19 fire at Arizona Public Service’s McMicken Energy Storage facility. Other notable incidents include a lithium battery fire and subsequent battery malfunction that led the Federal Aviation Administration in 2013 to ground Boeing’s entire 787 Dreamliner fleet. The next lithium battery fire can happen almost anywhere, anytime.

To safeguard against fire risks, ask lithium battery makers the questions about cell production and testing in this post. Battery buyers don’t have to wait for technology development or new regulations. They can bring about a new safety standard by demanding better safety data and buying lithium batteries only from OEMs that make the data available.

Questions to ask about cell production

Some online shoppers go to commerce platforms like Kickstarter for innovative products and products that may be available at a significant discount from an upstart manufacturer. When sourcing lithium batteries, you want to take the opposite approach. Instead of pursuing innovative products, look for proven products that have a long track record of consistent production. Instead of hunting for discounts from unknown suppliers, expect to pay fair value for a product that has completed a rigorous safety analysis and achieved an exceptionally low failure rate.

How many battery cells and battery packs does your supplier produce each year?

One lithium cell represents one data point. The more cells you produce, the more data you have. As such, the highest-volume producers have the most data on performance, thermal runaway, and failure.

For this reason, an OEM producing 10 million cells per year should have a better understanding of cell safety and performance. Large-volume manufacturers have probably seen every possible failure occur many times. By the same token, a small-volume manufacturer needs more time to analyze and understand cell failure. Buying cells from small-volume manufacturers may carry more risk.

What changes have been made to the battery cell and battery pack production process?

A consistent manufacturing process yields predictable safety and performance results. It’s plain to see that different cell materials bring about different cells. But different production equipment can affect safety characteristics just as much. Even if the materials and equipment stay the same, a manufacturer that relocates production may alter a host of environmental conditions and other variables that affect results. Changes in relative humidity, temperature range, and impurities in the air can impact safety characteristics of lithium cells. Differences in quality control and other processes introduced by a new manufacturing technician crew can also have an effect. All these changes should be understood and quantified in a prudent lithium battery safety analysis.

How does your supplier handle material acceptance and storage?

Even if everything goes right during the manufacturing process, pre-production material acceptance and storage can affect lithium battery safety. About five years ago, a global supplier of solar inverters experienced a series of product failures after electronic circuit boards had been stored in the wrong warehouse and exposed to moisture. Once product assembly was complete and the inverters were energized, a short circuit on the boards caused a fire and led to quite a bit of property damage. While moisture can also affect battery cell safety, so can other environmental conditions, such as air impurities and particulate matter.

It’s not easy to perform a safety analysis that identifies failure points for battery cells. To test how moisture affects safety, you would have to take identical cells and store one of the cell materials in an environment that gets wetter in small increments until you find a statistically relevant number of failures. Then you would have to repeat the process with incremental changes in temperature, dust, and other variables. Testing would require a lot of cells, a lot of minor changes in cell processing, a lot of time, and a lot of analysis. And it would all have to be done without vastly increasing cell production costs.

What are the failure rates for your supplier’s battery cells and battery packs?

In the absence of industry-wide standards, contractors seeking assurances about product safety have their work cut out for them. First, they have to request failure rates and analysis from each of their suppliers. Then the manufacturers must provide the data. Next comes the subjective test. If the contractor feels comfortable with the risk, he or she can decide if the battery quality is adequate. Different contractors have different tolerance levels for quality. A contractor who installs one small system per year may not place a great deal of emphasis on quality. The chances of failure are small. However, contractors installing many large systems must pay more attention to quality. Their businesses depend on the successful operation of a much larger population of cells.

Consider an OEM with a 99.98 percent success rate for battery cells in the first three years of operation. That translates to a 0.02 percent failure rate. If a contractor installs 1,000,000 cells per year, the contractor can expect 600 cells to fail. [Multiply failure rate (0.0002) x annual production (1,000,000) x number of years (3).] This might be an unacceptable level of risk. On the other hand, if a contractor installs 10,000 cells per year, the contractor can expect 6 cells to fail. This level of risk might be no big deal, so long as those cell failures don’t propagate to the entire pack or the entire storage system.

Questions to ask about cell testing

We all know not to leave a fireplace unattended or a gas oven running when nobody’s home. We understand that doing so introduces a serious risk of fire. But how many people know the temperature threshold that is likely to cause a lithium battery to catch fire or explode? Before procuring lithium batteries, especially those that will be sited at a building where people live or work, be sure to understand the conditions that create lithium battery safety hazards. Safety hazards that start in a single battery cell can quickly spread to the battery pack and the entire energy storage system.

What are your supplier’s battery cell thermal runaway characteristics?

It’s important to understand how a battery cell responds to the conditions that can initiate a fire or an explosion. There are many ways to test lithium cells for these conditions. Some examples are the top nail test, where a nail of standard size is driven with standard force into the top of the battery, and the side nail test, where the same procedure is carried out with a battery lying on its side.

Other tests include the fast heat test, where a battery inside a control chamber is exposed to a rapid temperature increase; the slow heat test, where a battery is exposed to a slow temperature increase, and the overcharge test, where a fully charged battery stays connected to a power source and is continually charged.

What is the probability of thermal runaway for your battery cells?

With test results in hand, you can make reasonable predictions about how a battery will perform according to design specifications. Graph 1 shows how increased temperature leads to thermal runaway. While all five cells exhibit similar power generation as temperature increases, there is a notable difference in how close each cell comes to the failure point represented by the horizontal red line at 160°.

how increased temperature leads to battery cell thermal runaway

Graph 1

Graph 2 shows how constant temperature over time leads to thermal runaway. The battery cell depicted by this graph remains at very low risk of thermal runaway when temperature is held constant at 159°. But a 1° increase in constant temperature vastly increases the probability of thermal runaway. A 2° increase makes thermal runaway a near certainty.

how constant temperature over time leads to battery cell thermal runaway

Graph 2

One of the challenges when characterizing lithium cell failure is calculating at what temperature and over what duration a cell fails. Because the answer is different for each cell, we need to see how different the answers are. What if one cell failed after two hours at 60°, another cell failed after 5 hours at 190°, and a third cell failed after 3 hours at 250°? This data would be difficult to characterize. It seems like almost every temperature is dangerous and could lead to cell failure.

Now what if the data looked more like this? Cell 1 fails after two hours at 160°, Cell 2 fails after two hours at 161°, and Cell 3 fails after 1.5 hours at 162°. This data suggests that thermal runaway is consistent and predictable. If we can find consistent results, we know when failures occur and how to prevent failure by designing systems for lithium battery safety.

How does thermal runaway spread from cell to cell?

This is really a two-part question. For starters, let’s look at how thermal energy from a failed lithium cell gets distributed across neighboring cells. Do all neighboring cells get the same amount of energy from the failed cell? Does one cell get all the energy while the others get none? Do two cells get 90 percent of the energy? Next, let’s look at how much stress an initiator cell applies on neighboring cells. If a failed cell exposes neighboring cells to temperatures up to 120°, the risk of cell-to-cell propagation is low. The risk is much higher if a cell failure has a magnitude of 180°. If energy is distributed unevenly, we would want to know the magnitude of stress for each of the neighboring cells.

What are the conditions that lead an entire battery pack to catch fire?

If cell-to-cell propagation extends to one or two neighboring cells and stops, failure of the whole battery pack or battery module is unlikely. If cell-to-cell propagation extends to hundreds of neighboring cells, it’s far more likely that the entire pack will burn. By understanding when battery packs catch on fire and start heating up the neighboring packs, the system designer can plan for fire detection and suppression systems as required by NFPA 855 and UL 9540A to kick in as a last line of defense.

What are the conditions that lead an entire energy storage system to catch fire?

If a fire containment system fails to contain a fire within the energy storage system enclosure, the charging infrastructure for the batteries may also catch on fire. (Think of a car catching on fire while being pumped with fuel at the gas station. Fire can spread to the gas pump, then the entire gas station.) Once the charging infrastructure is on fire, the entire property, including its occupants, are at risk. In sum, one battery cell failure can lead to the destruction of an entire building and the loss of life.

Demand lithium battery safety data

A safe battery is a well-documented battery. Test data helps engineers, system integrators, system owners, and regulators make smart, effective business decisions. While data doesn’t eliminate risk, it does inform us of the risk. Therefore, data empowers us to make decisions on how to manage, contain, suppress, mitigate, or ignore it. By putting appropriate measures in place, we can reduce risk to an industry-acceptable level. Without test data, a battery might operate safely today, but we don’t know why. Then if conditions change and the battery is no longer safe, we won’t know how to mitigate the risk.

The industry can expect a steady supply of safe lithium batteries as soon as buyers make purchasing decisions conditional on access to safety data. There are many safe batteries on the market. But there are many more cheap, risky batteries on the market. The simple solution is to buy safe batteries—which might mean accepting a higher price.

Contractors should request data from OEMs or look for third-party evaluations from independent engineering (IE) reports or independent test laboratories. The data is not publicly available, or hard to find, which is a serious problem. UL, the “gold standard” in product safety, even has trouble gaining access to this sort of data. So one requirement in the UL 9540 standard is to capture thermal runaway data, even for batteries that pass all the tests. In other words, when a lithium battery goes for the UL 9540 test, the test lab will force the battery into thermal runaway and then document the results to help characterize lithium cell failure.

This post was written by Josh Weiner, Solar Expert Witness & Solar Engineering Expert. Mr. Weiner has been at the forefront of the solar energy industry for over 20 years and is an industry leader on solar-plus-storage engineering & design. Josh’s expertise spans both in-front of and behind-the-meter initiatives including residential, commercial, utility, grid-scale, and ev charging solar and storage applications.

When I saw this article about LG lithium-ion energy storage fires in Korea, I couldn’t help but think of the fires that PG&E is being held responsible for in California. Those fires have ultimately lead PG&E into bankruptcy and will inevitably increase energy costs to ratepayers.

It’s amazing how something as seemingly simple as a campfire, power line, or a 18650 lithium cell—about the size of a lipstick container–can cause so much damage to California, one of the wealthiest states in the world and PG&E, the largest utility in the state, and, of course to the loss of lives and homes.

Some of these hazards defy logic or at least expectations. When SepiSolar was providing technical due diligence and engineering review services to NRG Home Solar from 2014 – 2016, we came across residential projects on the East coast that had unexpected dangers. For example, there was a solar PV system installed on top of the garage where snow had piled up on the PV system. Some rain had turned that snow into a giant slab of hardened ice. When the ice slipped off the solar array, it crushed the car parked in the driveway–not dented, dinged, or scratched. It completely totaled the car. The homeowner told us “that’s exactly where my children play in the summertime.”

Having just become a father at the end of December 2018, I think it’s fair to say that safety cannot, should not, and will not ever be taken for granted on my watch.

Risks vs Benefits

I don’t mean to suggest that we ought to over-design, over-engineer, over-regulate, over-install, or somehow bullet-proof every single component or assembly in a traditional solar or storage system. That’s like saying “Since car accidents kill people, let’s require everyone to drive army-grade tanks down the street.” That line of thinking effectively kills an industry and becomes a zero-sum game. Instead, I would pose that taking risks is a part of life and is healthy for us, since taking risks and stepping outside our comfort zones is exactly how we grow, learn, and evolve.

The goal is to take calculated risks, or, alternatively, educated risks. What’s a calculated risk? It’s a risk that you’re aware you’re taking. The difference between educated risks and blind or reckless risks is awareness.

We then need to weigh those risks against the benefits in order to make effective decisions. After those decisions are made, we need to be ready to revisit them again soon because the learning process never stops. Assumptions will need to be revised, data recalculated, risks revisited, benefits re-weighed, and decisions re-evaluated. This is how we evolve and approach an ever-safer future, together.

So, let’s build some awareness, shall we? Let’s have a data-driven discussion about the fire risks associated with energy storage systems, and let’s turn our blind risks into calculated ones. Having helped build Green Charge Networks into a nationwide energy storage integrator (acquired by Engie in 2015), engineered solar and battery systems for over 10+ years, and having worked with utilities, UL, code officials, etc. on safety standards, I think I might have a thing or two to say about this subject.

Evaluate the Energy Storage Technology

To minimize risks in energy storage, perhaps the most obvious approach is to work with a technology that inherently works with chemicals and materials that have no fire risk associated with them. This is particularly difficult with batteries because when almost any battery is short-circuited, they instantly become a fire hazard. But that’s the nature of batteries – they can produce insanely high amounts of current, since the resistance in the battery circuit is governed by however fast (or slow) the chemicals involved can react with each other, allowing the free flow of electrons to accumulate. Of course, these chemicals are designed to react with each other in order to release electric charge. So, fire hazard is almost inherent in any battery (with at least 1 exception).

I love this side-by-side technology comparison authored by Fire Captain Matthew Paiss, a 22-year veteran of the San Jose Fire Department. Captain Paiss is the Fire Department’s subject matter expert on energy storage and is the IAFF primary representative to NFPA 70 (National Electrical Code) and NFPA 855 (Energy Storage System Standards), which has been incorporated into UL standards such as UL 9540. It was surprising and gratifying to know that there’s at least 1 technology that rises above the rest when it comes to safety.

Codes & Standards

There are a ton of uber-smart tradesmen, engineers, officials, and subject matter experts who love to wordsmith and craft codes and technical language (God love them!) in order to impose a minimal, universal set of health and safety standards designed to protect personal property and life. Some of these codes go all the way back to 1897, as is the case with the National Electrical Code, when electricity was thought of as a liquid! (Check out Leyden jars.)

Bottom line, let’s be sure to read and understand the modern codes thoroughly, including NFPA, NEC, UL, among others. Every word, comma, and comment were crafted with the care one would expect of a nationally applicable set of requirements, even if you disagree with many of them. It’s important to follow voltage, current, and sizing requirements, naturally. NEC 706, for instance, was just added to the NEC in the 2017 edition. That’s the first time batteries have been overhauled in the NEC since Article 480 was written back in the early 20^th century! Let’s expect this new code section to evolve with the times as more data becomes available and continue to think of these codes as a “minimal” set of safety standards that we can go above-and-beyond as necessary to ensure the safety of the systems we design and build.

Real-time Data

While codes and standards are important, one of their drawbacks is that they are slow to change. Technology and data often evolve faster than codes and policies. Because of this, it’s important to look at the data, stay up-to-date on the latest-and-greatest information available, and dynamically build this data into your systems as it becomes available. Basically, I’m advising you to read. Read articles, publications, journals, media newsletters, and absorb as much as possible to keep up-to-date.

For instance, now that the above Korean article has surfaced about LG battery fires, it’s imperative to find out the root cause failures that led to these hazards. There is much to learn from failure, thereby converting failure into learning opportunities (which perhaps negates the use of the term “failure” in the first place – nothing is a failure, so long as you learn something from it!). We don’t have to wait for new technologies or new codes to come out. Instead, let’s use the data right away in any or all systems that we may be using with LG batteries, or any battery, for that matter.

The first time I thought about the risks associated with batteries was when I heard that Boeing grounded the Dreamliner. Our Co-Founder and CEO of Green Charge Networks at the time was a retired Boeing executive, so this naturally caught our attention. Wikipedia does a decent job summing up that experience, and you can get the full investigative report here.

The general takeaway is that regulatory bodies, manufacturers, and engineers were not “up to snuff” on the risks associated with battery technology. To a great degree, as the above Korean article shows, we are still learning these risks. At our time at Green Charge Networks, we understood that this meant that the safe deployment of battery systems would largely rest on us, since codes, standards, products, and regulations were still too much in their infancy to support us.

Direct Experience and Training

Nothing prepares you for danger, uncertainty, or risk more than education, experience, and training. The more hands-on experience you have with a particular product or technology, the more you will understand its limitations, weaknesses, and risks. Understanding not only what and when a battery undergoes thermal runaway, but also the “how” can really help put battery risks into perspective. What I learn from this is that it’s not just the battery one should be cautious of, but also the environment the battery is in. For example, does the battery have a fire suppression system? Is the battery located near any buildings or structures that have no fire suppression?.

One time I dropped a wrench on an old golf cart battery, and it just so happened that the wrench landed perfectly on both positive and negative terminals simultaneously. It was the first time I saw metal turn bright red, orange, and then white, and eventually melting all over the battery. This was just a regular ol’ lead acid battery, so it was surprising to me that such an old battery could have such a great impact on something as solid and stiff as a wrench. Needless to say, I am very cautious around terminals of batteries, since most batteries cannot be inherently turned “off” (again, with some exceptions).

In a nutshell, if you’re working with lithium batteries, make sure to identify the risks and retire them as much as possible. For instance:

HVAC systems for lithium are not just there to support battery performance, but they are safety devices as well. Make sure they’re appropriately sized and adequate for the operating environment the batteries will be in.
Lithium batteries that get too hot can result in thermal runaway, and other types of hazards, aside from accelerated degradation of the cell capacities and efficiencies. Fire suppression systems are required with the appropriate cleaning agents.
Closely monitoring and isolating cells that are approaching their end-of-life is critical. Battery degradation not only leads to capacity loss, but also battery failure.

There are many other aspects to keep in mind, and nearly all are avoidable if you’re aware of them in the first place.

I strongly believe that lithium-ion battery systems will continue to grow and thrive in our new renewable energy world, but as the Korean article shows, there are risks. As engineers, it’s our responsibility to be aware of these risks, evaluate them, and to find the solutions that will decrease those risk and perhaps even eliminate them with new safety innovations.

SepiSearch

How to prevent a lithium battery fire