Charged EVs | Why EV charging breaks (and how testing labs prevent it)

• The EV charging industry may seem to be divided between hardware and software specialists, but the two are mutually interdependent, and must work together on various levels.

• BTC Power’s chargers incorporate three main categories of firmware, which enable the hardware to communicate with the vehicle and with the backend software that powers user authentication and payment processing.

• EV charging hardware must be tested for compatibility with vehicles and with backend software systems on an ongoing basis. BTC Power maintains a testing lab and a regular schedule of testing and re-testing.

Ensuring charging hardware and software compatibility: Q&A with BTC Power’s Bill Seamon

EV charging, like most modern technological endeavors, depends on a stack consisting of hardware, software and services, which work together to deliver the desired result. Some companies offer the full stack (e.g. Tesla, or the many companies that offer Charging as a Service), while others specialize in hardware (ABB, Tritium), and others stick to software (AMPECO, Driivz).

We’ve covered heaps of hardware manufacturers and scads of software providers, but when we look a little closer, we find that these are not mutually-exclusive categories. Charging hardware incorporates various kinds of software (firmware, low-level operating systems), and software companies have to test their wares with a wide range of hardware products on an ongoing basis, so sellers of the hard and the soft find themselves working together on various levels.

BTC Power, which describes itself as the second-largest US maker of EV chargers after Tesla, understands this symbiotic relationship as well as anyone. The company has integrated its chargers with more than 50 software systems, and maintains a large testing lab, where, the company says, “all the automakers” test their vehicles for compatibility.

Here at Charged, while we pride ourselves on delving into technical details of EV charging products, one of the reasons we focus on one-on-one interviews is that we love to hear real-world stories about how companies solve problems for their customers. When we approached BTC Power for an article about the company’s testing procedures, the PR people hooked us up with Senior Program Manager Bill Seamon because “he has the best customer stories.”

Charged: So you’re the man with the juicy stories from down in the trenches of charging?

Bill Seamon: Yeah. I was the “test and release” manager. Now I’m a program manager and I’m overseeing a backend migration project right now.

Charged: Backend migration? So, for example, if a charge point operator loses their backend software because the software provider pulls out of the market, you come to the rescue?

Bill Seamon: We’re also getting customers just changing backends from one company to another. Maybe they’re not getting support, I don’t know. But we’re getting a lot of that also.

Charged: I imagine most of your customers are either CPOs or fleet operators. Are there some other categories?

Bill Seamon: We have the whole range, from customers buying hundreds of chargers to customers buying ones and twos. We have the major charge providers; we have the CPOs that sell third-party to customers and help them install it; and then we have individual customers buying two, three, five chargers to put out in front of their convenience stores. We also have utility companies, school districts and hospitality companies who are our direct customers.

Charged: Those customers must have very different needs—I imagine a CPO or a big fleet operator is likely to be quite knowledgeable about charging, whereas a small customer probably has a lot of questions.

Bill Seamon: Yeah, some of them will buy the chargers from us, but then they’ll work with a backend provider that’ll help them install and run through the permit process. But It’s a whole new industry. I think everybody’s still trying to figure out the best way to do it.

Charged: There are a lot of charger companies out there and, well, chargers all do the same thing. How do you differentiate your products from your competitors?

Bill Seamon: Ours are a little more expensive, but we put a lot inside. I think safety is the biggest thing we bring to the table. These things are putting a lot of power through that cable, and heaven forbid somebody gets hurt charging their car. We have thermistors in the cable, in the head, and internal to the charger, and we’re monitoring those thermistors. There’s a big difference from charging a car in Arizona in August to charging a car in Minnesota in December. There’s a lot of internal software checks that are being done all the time while we’re charging the vehicle.

Another advantage we offer is stability. We are probably one of the few charger manufacturers that are profitable. We’re making money. Not a lot, but this is a new industry and it’s going to take several years for everything to shake out. But chances are we’re one of the few that are going to be around in 5, 10 years.

Charged: Everyone tells me there’s a shakeout going on, and it’s not over yet.

Bill Seamon: I spent 24 years at Western Digital in the disk drive market. Back then there were hundreds of disk drive manufacturers, and now there’s really two. The same type of thing will probably happen in the EV market. Same with the backend providers, the network providers.

Charged: You have integrated your chargers with more than 50 software systems. How do you test your hardware with all these different software products?

Bill Seamon: We do integration with the backend software to make sure our charger processes billing and credit cards. We have weekly meetings with the larger backend providers. We have a major backend provider coming this week for several days of testing on site.

We have contacts with makers of all the different credit card devices. We’ve had problems in the past where the device manufacturer will update the software without telling us, and the library that our software uses to talk to the credit card becomes incompatible with the new version. We have the capability to talk to our chargers over the air all over the world—we can update the software, do diagnostics, download logs for debugging. There’s some pretty sophisticated things going on inside that charger to make sure we can support it.

We also test with the car manufacturers. Here’s a story about one of the major European vehicle manufacturers. On the CCS2 connector there’s a button you press to release the connector. The vehicle senses that button press, it turns off and tells the charger to shut things down, then the vehicle releases the connector so we can pull it out.

And this major manufacturer never knew what that button was. When they sensed that button press, they did an emergency shutdown, which caused arcing in our internal power supply, because you can’t just shut down 300 kilowatts going through a connector. We’re like, “Guys, you’re killing us here. You can’t do that.” And they said, “We’ve never seen that button before. We didn’t know what to do.” The spec is pretty cryptic. My only thought was: Engineers that can’t design, write specs.

Charged: We’re talking about the Open Charge Point Protocol?

Bill Seamon: Yeah, and the connector spec and the Level 2 spec—it’s all tied together.

Charged: When a backend provider says they’re charger-agnostic, is that just as simple as being OCPP-compliant?

Bill Seamon: There are different implementations and different levels of OCPP. We meet most of those. But again, it’s the vehicle manufacturer interpreting the spec. We work with these backend providers, run some basic charging tests. I’ve got 15 different backend providers that we’re working with. The backend provider may say, “We need OCPP 1.6.” But there’s a lot of vendor-specific or customer-specific commands that can be implemented. The spec is the core, but there’s a lot of other stuff that gets put into that.

Charged: Your charger is not just a dumb piece of hardware that works with somebody else’s software. You’ve got your own software layers that have to interface with backend providers and all kinds of other software. How does all that fit together?

Bill Seamon: We basically have three major pieces of software. There’s a board in the dispenser that talks to the car and that does the CCS protocol. And we have several people working on that because it’s always changing. A new vehicle will come out, we’ll test with that vehicle. We find stuff. They find stuff.

Second, we have internal firmware that’s in the dispenser and in the tower. For our standalone charger, we have one set. But for installations that have dispensers, there’s a DC cable and CAN lines going from the dispensers to the tower. So, there’s firmware in the tower and firmware in the dispenser talking back and forth for sending power and shutting power off.

Third, there’s what we call the point-of-sale firmware. This is a Java application in the dispenser that provides the GUI interface to the customer, processes credit card transactions, and then talks to the backend provider to send all that information back and forth. Each credit card device works differently, and there are multiple credit card devices that customers want.

Another story—I got a call from a customer that had installed some of our old 50-kilowatt chargers, and they had to wait over a year before the power company gave them power to turn them on. When they turned them on, they found that they had these obsolete card readers that the backend provider couldn’t support. So, we were scrambling to try and see if our point-of-sale application could support that customer, because the backend provider couldn’t.

That brings up another point—dealing with these different backend providers, with the vehicle manufacturers, with the credit card companies, gives us wide knowledge of how to help these customers, which may be the owner of the charger or the person trying to charge his vehicle when he’s in the middle of nowhere. We have a team that works pretty much around the clock that’s tied into our customer support phone number.

If a software provider pulls out, and all these people are panicking and scrambling, they can turn to BTC Power and we’ll know how to help you as quickly as possible.

Charged: It sounds like there’s a certain amount of overlap between the frontend and the backend software. In your example of the credit card reader, the backend couldn’t support the credit card reader, but you found a way to support it.

Bill Seamon: Yeah, there’s different systems for credit card readers. In some cases, we talk to the credit card reader and to their payment network. There’s other companies where we don’t even talk to the credit card reader—it has a separate ethernet port, and when you swipe the card, that reader talks to their server and it knows that that reader is registered to a specific backend and customer. That customer’s backend does the payment, the backend server tells us it’s okay, and we start charging. That’s called an around-the-loop transaction. In other cases, we talk to the credit card reader directly, and in some cases, it’s in between. There’s a lot of different protocols involved.

Charged: So, in some cases, a particular function might be handled by your software, and in other cases it might be handled by the backend?

Bill Seamon: Yeah, it’s all specific based on the backend, the credit card reader and the charger, so there’s all that software going on inside our charger to make sure everything is communicating properly.

Charged: When you’re testing your hardware with different software and different vehicles, is there a standardized testing regimen, or some sort of a scorecard?

Bill Seamon: Well, there’s all different kinds of testing. Right now we’re working with OCA, which is a company that got a certification for OCPP 2.0, 1.6 and all that. They have a lab in Virginia and we have our chargers set up there permanently. We work with them to get OCPP 2.0 certification—they run the test, they send the report to OCA, OCA reviews the report and gives us our certification stickers. We run through that periodically. Different backend providers have their own test processes that we follow.

Charged: I hear a lot about microgrids. A large installation includes not only the charging hardware, but switchgear, transformers and possibly battery storage and/or onsite generation. Tell me a bit about how all that stuff fits together.

Bill Seamon: When our charger talks to the backend provider’s server, it also talks to our own monitoring server. For each customer, we have a monitoring server with its own internal protocol. When there’s a problem, we can react quicker than waiting for the customer to complain because we have a server talking to all our chargers. For that customer, we know what site they’re at, we know the charger serial number…all that information.

Also, we’re working with several large manufacturers to provide energy management. Say we’ve got 10 chargers at a site. They all have, let’s say, 350 kW capability, but we don’t have that much power at the site. If everybody plugs in at the same time, we’ll be reducing the power to all the vehicles so they can all use their chargers and manage that energy at that site based on their power company’s capabilities.

Charged: A subject I’ve been hearing a good bit about in connection with microgrids is cloud control versus local control. Amber Putignano at ABB told me that she sees a trend for more of the control functions to be handled on site, as opposed to everything going to the cloud and back. But Oren Halevi at Driivz, a software provider, pointed out that certain things, like authentication and payment processing, have to happen in the cloud.

Bill Seamon: Yeah, the problem is the chargers don’t talk to each other. They talk to our monitoring portal or to the backend provider, which is the cloud, so if you were to have something local, you would need a third connection for that charger to talk to a local server. I don’t see that happening right away because it would be redundant and extra cost. And today, your credit card transactions are all cloud-based. Our charger will record transactions, and we can keep local account information on it, so if we do lose the internet for a short period of time, then once the internet comes back, we would send all that information. But the payment is really dependent on the cloud anyhow, so you just have to have a reliable internet connection.

Charged: It must be a challenge to make sure new software works with older hardware that may still be in service. And as software and standards change, I guess you have to re-test with customers periodically.

Bill Seamon: In the lab, we have all our chargers on skids, with pigtails coming off of them. If somebody wants to test on an older model, we lift-truck them in, plug them in and begin testing with them within an hour.

For each customer we’ve tested with, we’ll re-test anywhere from every three weeks to every six months. We have a matrix of each version of code that we last tested with each vendor or backend provider. There may be new code coming along, but we don’t change that provider’s code until we test with them and get approval.

We have a software release notice process that we keep internally. We document the whole process so we can keep track of what we released for whom. We record the versions we last tested with Customer X on this device, this charger model, etc. Here’s the new versions, here’s the changes, here’s where they gave us authorization to do a pilot, to do the full rollout.

We have this firmware matrix that has all our customers, all the models. Right now it’s a big shared Excel spreadsheet, and we’ve probably got 400 lines of code in that spreadsheet to keep track of all these different customers. Once this industry consolidates, it’ll make my job a lot easier, to go from 400 to maybe 10.